[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:24:49 dist-test-slave-g04g krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:24:49 dist-test-slave-g04g krb5kdc[399](info): set up 2 sockets
Sep 26 07:24:49 dist-test-slave-g04g krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 07:24:50.707999 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:24:50 dist-test-slave-g04g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:24:50.826560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36575
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:50.932958 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:50.933265 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:50.933344 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:50.937307 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:24:50.937403 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:50.937441 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:50.937500 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:24:50.937537 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:24:50.942660 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:50.943712 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:50.944775 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:50.953477 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:50.953600 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:50.953495 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:50.953552 412 server_base.cc:1034] running on GCE node
I20260926 07:24:50.954198 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:50.955286 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:50.956452 412 hybrid_clock.cc:648] HybridClock initialized: now 1790407490956411 us; error 52 us; skew 500 ppm
Sep 26 07:24:50 dist-test-slave-g04g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:50.959235 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:24:50.960106 412 webserver.cc:460] Webserver started at http://127.0.98.62:36973/ using document root <none> and password file <none>
I20260926 07:24:50.960645 412 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:50.960772 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:50.960997 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:50.962991 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "c74d485c2cc64055ad2b763e2647fff0"
format_stamp: "Formatted at 2026-09-26 07:24:50 on dist-test-slave-g04g"
server_key: "5af9dae6a8a0981223abe1e86c302621"
server_key_iv: "3706f980c05704f6ac1e1881cf7b4152"
server_key_version: "encryptionkey@0"
I20260926 07:24:50.963627 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "c74d485c2cc64055ad2b763e2647fff0"
format_stamp: "Formatted at 2026-09-26 07:24:50 on dist-test-slave-g04g"
server_key: "5af9dae6a8a0981223abe1e86c302621"
server_key_iv: "3706f980c05704f6ac1e1881cf7b4152"
server_key_version: "encryptionkey@0"
I20260926 07:24:50.967517 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 07:24:50.970296 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:50.971553 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:24:50.971683 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "c74d485c2cc64055ad2b763e2647fff0"
format_stamp: "Formatted at 2026-09-26 07:24:50 on dist-test-slave-g04g"
server_key: "5af9dae6a8a0981223abe1e86c302621"
server_key_iv: "3706f980c05704f6ac1e1881cf7b4152"
server_key_version: "encryptionkey@0"
I20260926 07:24:50.971798 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:50.987504 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:50.990159 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:50.990391 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:50.997455 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36575
I20260926 07:24:50.997526 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36575 every 8 connection(s)
I20260926 07:24:50.998399 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:24:51.001238 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:51.002419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 412
I20260926 07:24:51.002538 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:24:51.003855 392 external_mini_cluster.cc:1428] Setting key 70d3f0cc828ab2380981cbc2461a0c0b
I20260926 07:24:51.007122 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0: Bootstrap starting.
I20260926 07:24:51.009578 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:51.010385 480 log.cc:824] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:24:51.012290 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0: No bootstrap required, opened a new log
I20260926 07:24:51.015688 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74d485c2cc64055ad2b763e2647fff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36575 } }
I20260926 07:24:51.015935 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:24:51.016016 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74d485c2cc64055ad2b763e2647fff0, State: Initialized, Role: FOLLOWER
I20260926 07:24:51.016461 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74d485c2cc64055ad2b763e2647fff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36575 } }
I20260926 07:24:51.016613 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:24:51.016697 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:24:51.016819 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:51.018011 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74d485c2cc64055ad2b763e2647fff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36575 } }
I20260926 07:24:51.018363 480 leader_election.cc:302] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c74d485c2cc64055ad2b763e2647fff0; no voters:
I20260926 07:24:51.018690 480 leader_election.cc:288] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:24:51.018757 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:24:51.018968 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [term 1 LEADER]: Becoming Leader. State: Replica: c74d485c2cc64055ad2b763e2647fff0, State: Running, Role: LEADER
I20260926 07:24:51.019353 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74d485c2cc64055ad2b763e2647fff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36575 } }
I20260926 07:24:51.020033 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:24:51.021051 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c74d485c2cc64055ad2b763e2647fff0. Latest consensus state: current_term: 1 leader_uuid: "c74d485c2cc64055ad2b763e2647fff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74d485c2cc64055ad2b763e2647fff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36575 } } }
I20260926 07:24:51.021178 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:51.021404 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c74d485c2cc64055ad2b763e2647fff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c74d485c2cc64055ad2b763e2647fff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36575 } } }
I20260926 07:24:51.021474 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0 [sys.catalog]: This master's current role is: LEADER
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:24:51.022490 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:24:51.026144 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:24:51.028565 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:51.006827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.0.98.62:36575)
0926 07:24:51.007249 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:51.007266 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:51.008359 (+ 1093us) server_negotiation.cc:407] Connection header received
0926 07:24:51.011568 (+ 3209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:51.011584 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:51.011889 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:51.012227 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:51.014347 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.015387 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.016867 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.017558 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.017740 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:51.022543 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:51.022550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:51.022564 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:51.022597 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:51.024800 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:51.025456 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:51.025462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:51.025468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:51.025606 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:51.026015 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:51.026019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:51.026021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:51.026547 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:51.026745 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:51.027456 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 07:24:51.027748 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":155,"threads_started":1}
I20260926 07:24:51.033450 494 catalog_manager.cc:1347] Generated new cluster ID: a1526dbd7b844e0197ab876cf7ec6073
I20260926 07:24:51.033514 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:24:51.046416 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:24:51.047358 494 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:24:51.054070 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c74d485c2cc64055ad2b763e2647fff0: Generated new TSK 0
I20260926 07:24:51.054726 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:24:51.086630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36575
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:51.198829 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:51.199122 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:51.199196 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:51.203204 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:51.203291 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:51.203379 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:24:51.208484 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36575
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:51.209522 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:51.210532 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:51.219767 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:51.219841 506 server_base.cc:1034] running on GCE node
W20260926 07:24:51.219838 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:51.219935 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:51.220479 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:51.221154 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:51.222343 506 hybrid_clock.cc:648] HybridClock initialized: now 1790407491222309 us; error 46 us; skew 500 ppm
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:51.225128 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:24:51.225919 506 webserver.cc:460] Webserver started at http://127.0.98.1:42989/ using document root <none> and password file <none>
I20260926 07:24:51.226428 506 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:51.226487 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:51.226645 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:51.228633 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "a1526b18ec18401f900cff101c298951"
format_stamp: "Formatted at 2026-09-26 07:24:51 on dist-test-slave-g04g"
server_key: "8635cf6652680688bee40048ee4cb807"
server_key_iv: "dc36a33eed30647c39244adc4f31d43f"
server_key_version: "encryptionkey@0"
I20260926 07:24:51.229125 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a1526b18ec18401f900cff101c298951"
format_stamp: "Formatted at 2026-09-26 07:24:51 on dist-test-slave-g04g"
server_key: "8635cf6652680688bee40048ee4cb807"
server_key_iv: "dc36a33eed30647c39244adc4f31d43f"
server_key_version: "encryptionkey@0"
I20260926 07:24:51.232733 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 07:24:51.234956 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:51.236160 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:24:51.236302 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "a1526b18ec18401f900cff101c298951"
format_stamp: "Formatted at 2026-09-26 07:24:51 on dist-test-slave-g04g"
server_key: "8635cf6652680688bee40048ee4cb807"
server_key_iv: "dc36a33eed30647c39244adc4f31d43f"
server_key_version: "encryptionkey@0"
I20260926 07:24:51.236415 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:51.256145 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:51.259220 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:51.259418 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:51.259991 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:24:51.260815 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:24:51.260890 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:51.260960 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:24:51.261013 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:51.269901 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35629
I20260926 07:24:51.269924 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35629 every 8 connection(s)
I20260926 07:24:51.270817 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:24:51.272759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 506
I20260926 07:24:51.272871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:24:51.273111 392 external_mini_cluster.cc:1428] Setting key ac1fe54c78422ca294ce2a62c466922d
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 07:24:51.285887 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:51.272752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37685 (local address 127.0.98.62:36575)
0926 07:24:51.272889 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:51.272894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:51.273739 (+ 845us) server_negotiation.cc:407] Connection header received
0926 07:24:51.275264 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:51.275267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:51.275317 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:51.275394 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:51.276968 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.277840 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.279067 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.279528 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.279665 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:51.282033 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:51.282043 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:51.282048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:51.282078 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:51.283629 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:51.284142 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:51.284144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:51.284146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:51.284214 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:51.284586 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:51.284588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:51.284589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:51.284751 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:51.284831 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:51.285519 (+ 688us) server_negotiation.cc:299] Negotiation successful
0926 07:24:51.285653 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:24:51.286588 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:51.273099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36575 (local address 127.0.98.1:37685)
0926 07:24:51.273586 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:51.273646 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:51.275009 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:51.275553 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:51.275560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:51.276101 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:51.276793 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:51.276816 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.278054 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.278057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:51.278922 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:51.278931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.279674 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.279677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:51.279818 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:51.280445 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:51.280460 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:51.281862 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:51.283749 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:51.283754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:51.283766 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:51.284046 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:51.284297 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:51.284300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:51.284301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:51.284499 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:51.284910 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:51.284916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:51.285235 (+ 319us) client_negotiation.cc:769] Sending connection context
0926 07:24:51.285450 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 07:24:51.285708 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":142,"threads_started":1}
I20260926 07:24:51.288422 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36575
I20260926 07:24:51.288719 635 heartbeater.cc:464] Registering TS with master...
I20260926 07:24:51.289237 635 heartbeater.cc:511] Master 127.0.98.62:36575 requested a full tablet report, sending...
I20260926 07:24:51.290990 444 ts_manager.cc:194] Registered new tserver with Master: a1526b18ec18401f900cff101c298951 (127.0.98.1:35629)
I20260926 07:24:51.292269 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:24:51.302011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36575
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:51.423053 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:51.423310 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:51.423384 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:51.427904 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:51.427961 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:51.428095 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:24:51.433596 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36575
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:51.434599 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:51.435603 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:24:51.445182 640 server_base.cc:1034] running on GCE node
W20260926 07:24:51.445137 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:51.445107 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:51.445117 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:51.445773 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:51.446372 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:51.447530 640 hybrid_clock.cc:648] HybridClock initialized: now 1790407491447516 us; error 28 us; skew 500 ppm
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:51.450263 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:24:51.451162 640 webserver.cc:460] Webserver started at http://127.0.98.2:35721/ using document root <none> and password file <none>
I20260926 07:24:51.451713 640 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:51.451802 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:51.452003 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:51.453966 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "8d292c82424546d487749fb138075278"
format_stamp: "Formatted at 2026-09-26 07:24:51 on dist-test-slave-g04g"
server_key: "60ba15500015dc6a8561b288f2a4dde7"
server_key_iv: "27d7a635335f20275ac2e9c7b7eae417"
server_key_version: "encryptionkey@0"
I20260926 07:24:51.454543 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8d292c82424546d487749fb138075278"
format_stamp: "Formatted at 2026-09-26 07:24:51 on dist-test-slave-g04g"
server_key: "60ba15500015dc6a8561b288f2a4dde7"
server_key_iv: "27d7a635335f20275ac2e9c7b7eae417"
server_key_version: "encryptionkey@0"
I20260926 07:24:51.458159 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 07:24:51.460448 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:51.461686 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:24:51.461836 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "8d292c82424546d487749fb138075278"
format_stamp: "Formatted at 2026-09-26 07:24:51 on dist-test-slave-g04g"
server_key: "60ba15500015dc6a8561b288f2a4dde7"
server_key_iv: "27d7a635335f20275ac2e9c7b7eae417"
server_key_version: "encryptionkey@0"
I20260926 07:24:51.461953 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:51.496269 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:51.499269 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:51.499480 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:51.500078 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:24:51.500911 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:24:51.500988 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:51.501073 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:24:51.501129 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:51.511439 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32935
I20260926 07:24:51.511487 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32935 every 8 connection(s)
I20260926 07:24:51.512349 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:24:51.519018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 640
I20260926 07:24:51.519193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:24:51.519459 392 external_mini_cluster.cc:1428] Setting key 4a903f7a2a3ff640af4b98a2d88ef7cd
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:24:51.526110 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:51.514080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41425 (local address 127.0.98.62:36575)
0926 07:24:51.514211 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:51.514215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:51.515040 (+ 825us) server_negotiation.cc:407] Connection header received
0926 07:24:51.516340 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:51.516342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:51.516386 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:51.516457 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:51.517561 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.518057 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.519169 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.519506 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.519613 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:51.522383 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:51.522388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:51.522393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:51.522425 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:51.523965 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:51.524565 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:51.524568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:51.524569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:51.524638 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:51.525008 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:51.525010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:51.525012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:51.525196 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:51.525308 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:51.525909 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 07:24:51.525999 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:24:51.527387 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:51.514432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36575 (local address 127.0.98.2:41425)
0926 07:24:51.514868 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:51.514909 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:51.516151 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:51.516605 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:51.516611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:51.516996 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:51.517441 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:51.517453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.518164 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.518167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:51.519017 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:51.519025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.519661 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.519665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:51.519860 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:51.520463 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:51.520481 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:51.522190 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:51.524074 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:51.524083 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:51.524100 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:51.524461 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:51.524722 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:51.524728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:51.524733 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:51.524916 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:51.525324 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:51.525334 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:51.525653 (+ 319us) client_negotiation.cc:769] Sending connection context
0926 07:24:51.525879 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 07:24:51.526235 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":105,"threads_started":1}
I20260926 07:24:51.528506 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36575
I20260926 07:24:51.528754 769 heartbeater.cc:464] Registering TS with master...
I20260926 07:24:51.529224 769 heartbeater.cc:511] Master 127.0.98.62:36575 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 07:24:51.530241 444 ts_manager.cc:194] Registered new tserver with Master: 8d292c82424546d487749fb138075278 (127.0.98.2:32935)
I20260926 07:24:51.530808 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:24:51.546715 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36575
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:51.659988 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:51.660290 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:51.660382 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:51.664407 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:51.664491 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:51.664680 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:24:51.669871 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36575
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:51.670890 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:51.671941 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:51.682379 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:51.682379 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:51.682437 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:51.682602 774 server_base.cc:1034] running on GCE node
I20260926 07:24:51.683145 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:51.683749 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:51.684904 774 hybrid_clock.cc:648] HybridClock initialized: now 1790407491684884 us; error 43 us; skew 500 ppm
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:51.687618 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:24:51.688408 774 webserver.cc:460] Webserver started at http://127.0.98.3:44821/ using document root <none> and password file <none>
I20260926 07:24:51.688951 774 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:51.689007 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:51.689210 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:51.691265 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "7b21ff4c39884dd4af27e8129df73810"
format_stamp: "Formatted at 2026-09-26 07:24:51 on dist-test-slave-g04g"
server_key: "bac41bec39a509070a07d38bc5b33c92"
server_key_iv: "0df0e19a7352ed3a304aca6cc259bf92"
server_key_version: "encryptionkey@0"
I20260926 07:24:51.691851 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7b21ff4c39884dd4af27e8129df73810"
format_stamp: "Formatted at 2026-09-26 07:24:51 on dist-test-slave-g04g"
server_key: "bac41bec39a509070a07d38bc5b33c92"
server_key_iv: "0df0e19a7352ed3a304aca6cc259bf92"
server_key_version: "encryptionkey@0"
I20260926 07:24:51.695477 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:24:51.697873 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:51.699244 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:24:51.699370 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "7b21ff4c39884dd4af27e8129df73810"
format_stamp: "Formatted at 2026-09-26 07:24:51 on dist-test-slave-g04g"
server_key: "bac41bec39a509070a07d38bc5b33c92"
server_key_iv: "0df0e19a7352ed3a304aca6cc259bf92"
server_key_version: "encryptionkey@0"
I20260926 07:24:51.699479 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:51.721475 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:51.724627 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:51.724834 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:51.725401 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:24:51.726217 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:24:51.726293 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:51.726363 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:24:51.726409 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:51.736364 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34909
I20260926 07:24:51.736439 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34909 every 8 connection(s)
I20260926 07:24:51.737273 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:24:51.742533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 774
I20260926 07:24:51.742677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:24:51.742969 392 external_mini_cluster.cc:1428] Setting key 90ee31c6138f232d202df9a1ef9916b8
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:24:51.750896 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:51.739124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41719 (local address 127.0.98.62:36575)
0926 07:24:51.739274 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:51.739279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:51.740171 (+ 892us) server_negotiation.cc:407] Connection header received
0926 07:24:51.741595 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:51.741599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:51.741655 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:51.741752 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:51.743006 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.743677 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.745006 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.745235 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.745345 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:51.747694 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:51.747697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:51.747699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:51.747722 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:51.749077 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:51.749539 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:51.749542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:51.749543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:51.749617 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:51.749920 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:51.749922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:51.749923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:51.750082 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:51.750173 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:51.750687 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 07:24:51.750784 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 07:24:51.751739 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:51.739548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36575 (local address 127.0.98.3:41719)
0926 07:24:51.740035 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:51.740077 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:51.741391 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:51.741943 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:51.741951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:51.742296 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:51.742867 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:51.742882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.743819 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.743822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:51.744883 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:51.744892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.745375 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.745379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:51.745548 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:51.746037 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:51.746053 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:51.747543 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:51.749187 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:51.749191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:51.749203 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:51.749443 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:51.749704 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:51.749706 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:51.749708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:51.749841 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:51.750179 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:51.750184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:51.750424 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 07:24:51.750633 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 07:24:51.750950 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":118,"threads_started":1}
I20260926 07:24:51.752833 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36575
I20260926 07:24:51.753028 903 heartbeater.cc:464] Registering TS with master...
I20260926 07:24:51.753427 903 heartbeater.cc:511] Master 127.0.98.62:36575 requested a full tablet report, sending...
I20260926 07:24:51.754370 444 ts_manager.cc:194] Registered new tserver with Master: 7b21ff4c39884dd4af27e8129df73810 (127.0.98.3:34909)
I20260926 07:24:51.754856 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41719
I20260926 07:24:51.757090 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 07:24:51.783246 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:51.774206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57964 (local address 127.0.98.1:35629)
0926 07:24:51.774448 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:51.774472 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:51.774569 (+ 97us) server_negotiation.cc:407] Connection header received
0926 07:24:51.774687 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:51.774692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:51.774904 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:51.775078 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:51.775790 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.776602 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.777409 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.777764 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.777843 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:51.780107 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:51.780114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:51.780126 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:51.780148 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:51.781763 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:51.782131 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:51.782135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:51.782136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:51.782208 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:51.782422 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:51.782424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:51.782425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:51.782631 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:51.782791 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:51.782915 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 07:24:51.783086 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
W20260926 07:24:51.784186 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57964
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 07:24:51.809646 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:51.801275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.0.98.1:35629)
0926 07:24:51.801446 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:51.801451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:51.801575 (+ 124us) server_negotiation.cc:407] Connection header received
0926 07:24:51.801764 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:51.801767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:51.801808 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:51.801894 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:51.802587 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.803215 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.804091 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:51.804337 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:51.804418 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:51.806713 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:51.806716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:51.806718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:51.806740 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:51.808028 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:51.808466 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:51.808474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:51.808475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:51.808545 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:51.808819 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:51.808824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:51.808826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:51.808987 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:51.809123 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:51.809363 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 07:24:51.809483 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:24:51.810662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 506
I20260926 07:24:51.816795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 640
I20260926 07:24:51.824440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 774
I20260926 07:24:51.829388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 412
2026-09-26T07:24:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2165 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[917](info): set up 2 sockets
Sep 26 07:24:51 dist-test-slave-g04g krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 07:24:52.870898 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:24:52 dist-test-slave-g04g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:24:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:24:52.998749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45501
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:53.105763 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:53.106017 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:53.106056 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:53.110095 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:24:53.110165 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:53.110186 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:53.110203 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:24:53.110260 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:24:53.115559 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:53.116561 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:53.117553 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:53.126003 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:53.126020 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:53.126010 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:53.126343 930 server_base.cc:1034] running on GCE node
I20260926 07:24:53.126881 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:53.128041 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:53.129231 930 hybrid_clock.cc:648] HybridClock initialized: now 1790407493129202 us; error 49 us; skew 500 ppm
Sep 26 07:24:53 dist-test-slave-g04g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:53.132093 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:24:53.132957 930 webserver.cc:460] Webserver started at http://127.0.98.62:42339/ using document root <none> and password file <none>
I20260926 07:24:53.133504 930 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:53.133589 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:53.133814 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:53.135881 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "05ec64010e15404fa5fbcd10401bba03"
format_stamp: "Formatted at 2026-09-26 07:24:53 on dist-test-slave-g04g"
server_key: "0f4b83b514b1f6bf224b1b124c4d38f1"
server_key_iv: "06dbe0264d5fd68cc5019f1bc078d14c"
server_key_version: "encryptionkey@0"
I20260926 07:24:53.136435 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "05ec64010e15404fa5fbcd10401bba03"
format_stamp: "Formatted at 2026-09-26 07:24:53 on dist-test-slave-g04g"
server_key: "0f4b83b514b1f6bf224b1b124c4d38f1"
server_key_iv: "06dbe0264d5fd68cc5019f1bc078d14c"
server_key_version: "encryptionkey@0"
I20260926 07:24:53.139914 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:24:53.142270 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:53.143566 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:24:53.143699 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "05ec64010e15404fa5fbcd10401bba03"
format_stamp: "Formatted at 2026-09-26 07:24:53 on dist-test-slave-g04g"
server_key: "0f4b83b514b1f6bf224b1b124c4d38f1"
server_key_iv: "06dbe0264d5fd68cc5019f1bc078d14c"
server_key_version: "encryptionkey@0"
I20260926 07:24:53.143803 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:53.165810 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:53.168488 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:53.168679 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:53.176033 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45501
I20260926 07:24:53.176106 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45501 every 8 connection(s)
I20260926 07:24:53.177006 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:24:53.179761 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:53.184516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 930
I20260926 07:24:53.184641 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:24:53.184891 392 external_mini_cluster.cc:1428] Setting key 2561a99f3e9bdc9508613138666712db
I20260926 07:24:53.185444 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03: Bootstrap starting.
I20260926 07:24:53.187738 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:53.188565 998 log.cc:824] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03: Log is configured to *not* fsync() on all Append() calls
I20260926 07:24:53.190490 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03: No bootstrap required, opened a new log
Sep 26 07:24:53 dist-test-slave-g04g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:24:53.193953 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ec64010e15404fa5fbcd10401bba03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45501 } }
I20260926 07:24:53.194186 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:24:53.194271 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ec64010e15404fa5fbcd10401bba03, State: Initialized, Role: FOLLOWER
I20260926 07:24:53.194753 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ec64010e15404fa5fbcd10401bba03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45501 } }
I20260926 07:24:53.194923 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:24:53.194998 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:24:53.195143 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:53.199530 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:53.186417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.0.98.62:45501)
0926 07:24:53.186899 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:53.186915 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:53.186951 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:24:53.187627 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:53.187654 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:53.188044 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:53.189119 (+ 1075us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:53.189551 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:53.190512 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:53.191477 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:53.191962 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:53.192152 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:53.194818 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:53.194824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:53.194840 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:53.194872 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:53.196844 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:53.197257 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:53.197262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:53.197266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:53.197369 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:53.197714 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:53.197716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:53.197718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:53.198082 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:53.198227 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:53.198476 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 07:24:53.198725 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":177,"threads_started":1}
I20260926 07:24:53.215597 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ec64010e15404fa5fbcd10401bba03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45501 } }
I20260926 07:24:53.216046 998 leader_election.cc:302] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05ec64010e15404fa5fbcd10401bba03; no voters:
I20260926 07:24:53.216418 998 leader_election.cc:288] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:24:53.216662 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:24:53.216852 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [term 1 LEADER]: Becoming Leader. State: Replica: 05ec64010e15404fa5fbcd10401bba03, State: Running, Role: LEADER
I20260926 07:24:53.217190 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ec64010e15404fa5fbcd10401bba03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45501 } }
I20260926 07:24:53.217891 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:24:53.218403 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05ec64010e15404fa5fbcd10401bba03. Latest consensus state: current_term: 1 leader_uuid: "05ec64010e15404fa5fbcd10401bba03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ec64010e15404fa5fbcd10401bba03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45501 } } }
I20260926 07:24:53.218523 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:53.218812 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05ec64010e15404fa5fbcd10401bba03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ec64010e15404fa5fbcd10401bba03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45501 } } }
I20260926 07:24:53.218922 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:53.218927 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:24:53.222312 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:24:53.227593 1008 catalog_manager.cc:1347] Generated new cluster ID: 2a0d72325f9f4d02bfdb0d8fc675d116
I20260926 07:24:53.227665 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:24:53.250597 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:24:53.251735 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:24:53.266753 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05ec64010e15404fa5fbcd10401bba03: Generated new TSK 0
I20260926 07:24:53.267711 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260926 07:24:54.272150 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.998s user 0.000s sys 0.004s
I20260926 07:24:54.272671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45501
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:54.384337 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:54.384634 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:54.384681 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:54.388616 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:54.388667 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:54.388725 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:24:54.393865 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45501
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:54.394878 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:54.395990 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:54.406175 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:54.406152 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:54.406139 1024 server_base.cc:1034] running on GCE node
W20260926 07:24:54.406237 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:54.406811 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:54.407617 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:54.408831 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790407494408776 us; error 65 us; skew 500 ppm
Sep 26 07:24:54 dist-test-slave-g04g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:54.417124 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:24:54.418215 1024 webserver.cc:460] Webserver started at http://127.0.98.1:38707/ using document root <none> and password file <none>
I20260926 07:24:54.418874 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:54.418978 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:54.419332 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:54.421618 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "c20d3ac70fb046f8a1381f5156e64d56"
format_stamp: "Formatted at 2026-09-26 07:24:54 on dist-test-slave-g04g"
server_key: "35a090a1580c8aa95d79527c90a8c85b"
server_key_iv: "612e5a603b2e34d309cf519ca1196a32"
server_key_version: "encryptionkey@0"
I20260926 07:24:54.422235 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c20d3ac70fb046f8a1381f5156e64d56"
format_stamp: "Formatted at 2026-09-26 07:24:54 on dist-test-slave-g04g"
server_key: "35a090a1580c8aa95d79527c90a8c85b"
server_key_iv: "612e5a603b2e34d309cf519ca1196a32"
server_key_version: "encryptionkey@0"
I20260926 07:24:54.426235 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:24:54.428973 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:54.430388 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:24:54.430542 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "c20d3ac70fb046f8a1381f5156e64d56"
format_stamp: "Formatted at 2026-09-26 07:24:54 on dist-test-slave-g04g"
server_key: "35a090a1580c8aa95d79527c90a8c85b"
server_key_iv: "612e5a603b2e34d309cf519ca1196a32"
server_key_version: "encryptionkey@0"
I20260926 07:24:54.430660 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:54.450124 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:54.453469 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:54.453694 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:54.454320 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:24:54.455211 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:24:54.455300 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:54.455394 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:24:54.455446 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:54.464560 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39225
I20260926 07:24:54.464599 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39225 every 8 connection(s)
I20260926 07:24:54.465504 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:24:54.469125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 1024
I20260926 07:24:54.469224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:24:54.469477 392 external_mini_cluster.cc:1428] Setting key 1f8aba8b7226a08377537856ba82e271
Sep 26 07:24:54 dist-test-slave-g04g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 07:24:54.482445 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:54.467976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40125 (local address 127.0.98.62:45501)
0926 07:24:54.468229 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:54.468233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:54.469079 (+ 846us) server_negotiation.cc:407] Connection header received
0926 07:24:54.471077 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:54.471083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:54.471139 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:54.471237 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:54.472510 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:54.473194 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:54.474591 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:54.474815 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:54.474921 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:54.477917 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:54.477924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:54.477930 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:54.477969 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:54.480020 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:54.480590 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:54.480596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:54.480608 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:54.480714 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:54.481176 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:54.481182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:54.481187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:54.481446 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:54.481560 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:54.482109 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 07:24:54.482247 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
I20260926 07:24:54.483327 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:54.468388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45501 (local address 127.0.98.1:40125)
0926 07:24:54.468899 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:54.468945 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:54.470755 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:54.471416 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:54.471423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:54.471800 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:54.472305 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:54.472319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:54.473326 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:54.473331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:54.474463 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:54.474473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:54.474988 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:54.474995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:54.475278 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:54.475934 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:54.475961 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:54.477729 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:54.480167 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:54.480172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:54.480184 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:54.480477 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:54.480889 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:54.480892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:54.480894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:54.481047 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:54.481571 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:54.481580 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:54.481834 (+ 254us) client_negotiation.cc:769] Sending connection context
0926 07:24:54.482060 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 07:24:54.482384 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":135,"threads_started":1}
I20260926 07:24:54.484625 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45501
I20260926 07:24:54.484900 1153 heartbeater.cc:464] Registering TS with master...
I20260926 07:24:54.485402 1153 heartbeater.cc:511] Master 127.0.98.62:45501 requested a full tablet report, sending...
I20260926 07:24:54.486922 962 ts_manager.cc:194] Registered new tserver with Master: c20d3ac70fb046f8a1381f5156e64d56 (127.0.98.1:39225)
I20260926 07:24:54.488085 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260926 07:24:55.440653 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.969s user 0.002s sys 0.001s
I20260926 07:24:55.441359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45501
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:24:55.490301 1153 heartbeater.cc:503] Master 127.0.98.62:45501 was elected leader, sending a full tablet report...
W20260926 07:24:55.556166 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:55.556471 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:55.556547 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:55.561064 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:55.561160 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:55.561261 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:24:55.566534 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45501
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:55.567651 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:55.568674 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:24:55.578362 1159 server_base.cc:1034] running on GCE node
W20260926 07:24:55.578308 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:55.578311 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:55.578305 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:55.578949 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:55.579618 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:55.580813 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790407495580798 us; error 28 us; skew 500 ppm
Sep 26 07:24:55 dist-test-slave-g04g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:55.584091 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:24:55.584970 1159 webserver.cc:460] Webserver started at http://127.0.98.2:35487/ using document root <none> and password file <none>
I20260926 07:24:55.585546 1159 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:55.585650 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:55.585891 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:55.588069 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "56b785e3f90148758a1a7f5832d44090"
format_stamp: "Formatted at 2026-09-26 07:24:55 on dist-test-slave-g04g"
server_key: "39079a020a82929204ac5c4b0dcaec27"
server_key_iv: "e9eb33c4d3cc12c071a0450cb53b97fe"
server_key_version: "encryptionkey@0"
I20260926 07:24:55.588637 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "56b785e3f90148758a1a7f5832d44090"
format_stamp: "Formatted at 2026-09-26 07:24:55 on dist-test-slave-g04g"
server_key: "39079a020a82929204ac5c4b0dcaec27"
server_key_iv: "e9eb33c4d3cc12c071a0450cb53b97fe"
server_key_version: "encryptionkey@0"
I20260926 07:24:55.592326 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:24:55.594652 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:55.595850 1159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:24:55.595978 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "56b785e3f90148758a1a7f5832d44090"
format_stamp: "Formatted at 2026-09-26 07:24:55 on dist-test-slave-g04g"
server_key: "39079a020a82929204ac5c4b0dcaec27"
server_key_iv: "e9eb33c4d3cc12c071a0450cb53b97fe"
server_key_version: "encryptionkey@0"
I20260926 07:24:55.596084 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:55.611193 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:55.614039 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:55.614238 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:55.614822 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:24:55.615697 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:24:55.615768 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:55.615849 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:24:55.615902 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:55.625212 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44441
I20260926 07:24:55.625243 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44441 every 8 connection(s)
I20260926 07:24:55.626121 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:24:55.627808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 1159
I20260926 07:24:55.627925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:24:55.628198 392 external_mini_cluster.cc:1428] Setting key 132db02820a8b8b82e86766127e0c60d
Sep 26 07:24:55 dist-test-slave-g04g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 07:24:55.641927 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:55.628054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49569 (local address 127.0.98.62:45501)
0926 07:24:55.628287 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:55.628293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:55.629152 (+ 859us) server_negotiation.cc:407] Connection header received
0926 07:24:55.631024 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:55.631066 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:55.631124 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:55.631204 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:55.632714 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:55.633623 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:55.634965 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:55.635429 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:55.635551 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:55.638592 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:55.638599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:55.638605 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:55.638638 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:55.640181 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:55.640633 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:55.640635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:55.640637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:55.640712 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:55.641002 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:55.641004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:55.641005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:55.641170 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:55.641237 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:55.641701 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 07:24:55.641795 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":74,"threads_started":1}
I20260926 07:24:55.642710 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:55.628501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45501 (local address 127.0.98.2:49569)
0926 07:24:55.628992 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:55.629044 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:55.630794 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:55.631417 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:55.631430 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:55.631911 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:55.632543 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:55.632562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:55.633812 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:55.633815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:55.634853 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:55.634865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:55.635589 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:55.635596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:55.635829 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:55.636443 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:55.636468 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:55.638429 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:55.640284 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:55.640288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:55.640300 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:55.640548 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:55.640800 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:55.640803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:55.640804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:55.640928 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:55.641259 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:55.641264 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:55.641504 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 07:24:55.641656 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 07:24:55.641893 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":147,"threads_started":1}
I20260926 07:24:55.643955 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45501
I20260926 07:24:55.644153 1288 heartbeater.cc:464] Registering TS with master...
I20260926 07:24:55.644587 1288 heartbeater.cc:511] Master 127.0.98.62:45501 requested a full tablet report, sending...
I20260926 07:24:55.645533 962 ts_manager.cc:194] Registered new tserver with Master: 56b785e3f90148758a1a7f5832d44090 (127.0.98.2:44441)
I20260926 07:24:55.646106 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260926 07:24:56.293081 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.663s user 0.000s sys 0.004s
I20260926 07:24:56.293792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45501
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:56.403280 1294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:56.403591 1294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:56.403667 1294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:56.407698 1294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:56.407785 1294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:56.407881 1294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:24:56.413239 1294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45501
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:56.414369 1294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:56.415418 1294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:56.425242 1299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:56.425287 1300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:56.425527 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:56.426014 1294 server_base.cc:1034] running on GCE node
I20260926 07:24:56.426518 1294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:56.427213 1294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:56.428414 1294 hybrid_clock.cc:648] HybridClock initialized: now 1790407496428400 us; error 28 us; skew 500 ppm
Sep 26 07:24:56 dist-test-slave-g04g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:56.431309 1294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:24:56.432116 1294 webserver.cc:460] Webserver started at http://127.0.98.3:41647/ using document root <none> and password file <none>
I20260926 07:24:56.432633 1294 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:56.432688 1294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:56.432849 1294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:56.434943 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "38c43e829c5e4813ae1ed3344188ff22"
format_stamp: "Formatted at 2026-09-26 07:24:56 on dist-test-slave-g04g"
server_key: "c88a953ea9177d53776710c22edb50d4"
server_key_iv: "bc123a3a85ba297e5875694e20546704"
server_key_version: "encryptionkey@0"
I20260926 07:24:56.435509 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "38c43e829c5e4813ae1ed3344188ff22"
format_stamp: "Formatted at 2026-09-26 07:24:56 on dist-test-slave-g04g"
server_key: "c88a953ea9177d53776710c22edb50d4"
server_key_iv: "bc123a3a85ba297e5875694e20546704"
server_key_version: "encryptionkey@0"
I20260926 07:24:56.439128 1294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:24:56.441530 1309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:56.442682 1294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:24:56.442819 1294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "38c43e829c5e4813ae1ed3344188ff22"
format_stamp: "Formatted at 2026-09-26 07:24:56 on dist-test-slave-g04g"
server_key: "c88a953ea9177d53776710c22edb50d4"
server_key_iv: "bc123a3a85ba297e5875694e20546704"
server_key_version: "encryptionkey@0"
I20260926 07:24:56.442931 1294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:56.458843 1294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:56.462249 1294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:56.462458 1294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:56.463114 1294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:24:56.463919 1294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:24:56.463989 1294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:56.464057 1294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:24:56.464109 1294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:56.474390 1294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41533
I20260926 07:24:56.474423 1422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41533 every 8 connection(s)
I20260926 07:24:56.475282 1294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:24:56.480078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 1294
I20260926 07:24:56.480208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:24:56.480475 392 external_mini_cluster.cc:1428] Setting key e2a0bf14833d57795d4d3ae804f17afe
Sep 26 07:24:56 dist-test-slave-g04g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:24:56.489545 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.477100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46947 (local address 127.0.98.62:45501)
0926 07:24:56.477432 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:56.477435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:56.478029 (+ 594us) server_negotiation.cc:407] Connection header received
0926 07:24:56.479446 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:56.479450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:56.479510 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:56.479625 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:56.480898 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.481443 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.482811 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.483080 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.483195 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.485815 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:56.485818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:56.485821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:56.485846 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:56.487502 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:56.488016 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:56.488019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:56.488021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:56.488106 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:56.488443 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:56.488448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:56.488450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:56.488638 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:56.488773 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:56.489255 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 07:24:56.489376 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260926 07:24:56.490337 1426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.477473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45501 (local address 127.0.98.3:46947)
0926 07:24:56.477884 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:56.477924 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:56.479242 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:56.479793 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:56.479800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:56.480172 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:56.480748 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:56.480764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.481580 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.481583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:56.482689 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:56.482698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.483228 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.483232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:56.483418 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.484061 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:56.484077 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:56.485654 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:56.487619 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:56.487626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:56.487640 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:56.487896 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:56.488198 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:56.488202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:56.488204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:56.488346 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:56.488770 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:56.488776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:56.489018 (+ 242us) client_negotiation.cc:769] Sending connection context
0926 07:24:56.489203 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 07:24:56.489537 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":109,"threads_started":1}
I20260926 07:24:56.491463 1423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45501
I20260926 07:24:56.491698 1423 heartbeater.cc:464] Registering TS with master...
I20260926 07:24:56.492156 1423 heartbeater.cc:511] Master 127.0.98.62:45501 requested a full tablet report, sending...
I20260926 07:24:56.493090 961 ts_manager.cc:194] Registered new tserver with Master: 38c43e829c5e4813ae1ed3344188ff22 (127.0.98.3:41533)
I20260926 07:24:56.493791 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46947
I20260926 07:24:56.494352 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:24:56.505131 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.497682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.0.98.62:45501)
0926 07:24:56.497850 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:56.497855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:56.497997 (+ 142us) server_negotiation.cc:407] Connection header received
0926 07:24:56.498146 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:56.498149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:56.498192 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:56.498278 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:56.498970 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.499531 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.500449 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.500670 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.500793 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.502079 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:56.502083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:56.502085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:56.502114 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:56.503482 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:56.503897 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:56.503902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:56.503904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:56.503991 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:56.504260 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:56.504264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:56.504266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:56.504471 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:56.504617 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:56.504850 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 07:24:56.504971 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:24:56.510251 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:24:56.513190 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:24:56.527611 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.521677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39225 (local address 127.0.0.1:40032)
0926 07:24:56.522036 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:56.522112 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:56.522247 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:56.522857 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:56.522864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:56.522888 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:24:56.523064 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:56.523073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.524524 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.524528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:56.526008 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:56.526018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.527133 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.527136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:56.527245 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.527284 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:24:56.527394 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:24:56.527499 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":32,"thread_start_us":105,"threads_started":1}
I20260926 07:24:56.528821 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.521863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.0.98.1:39225)
0926 07:24:56.522197 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:56.522204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:56.522228 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:24:56.522438 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:56.522445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:56.522666 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:56.522857 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:24:56.523232 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.524362 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.526125 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.526963 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.527147 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.527302 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:56.528544 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0926 07:24:56.528645 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":63,"threads_started":1}
I20260926 07:24:56.528892 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.522138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41533 (local address 127.0.0.1:38472)
0926 07:24:56.522511 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:56.522527 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:56.522641 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:56.524125 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:56.524129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:56.524145 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:24:56.524251 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:56.524257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.525624 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.525628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:56.526789 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:56.526798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.528345 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.528349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:56.528549 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.528603 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 07:24:56.528678 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:24:56.528767 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":70,"thread_start_us":90,"threads_started":1}
I20260926 07:24:56.529026 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.522859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.0.98.2:44441)
0926 07:24:56.523261 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:56.523269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:56.523904 (+ 635us) server_negotiation.cc:407] Connection header received
0926 07:24:56.523966 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:56.523969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:56.524150 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:56.524283 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:24:56.524621 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.525319 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.527411 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.528209 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.528333 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.528465 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:56.528767 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 07:24:56.528877 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":135,"threads_started":1}
I20260926 07:24:56.529285 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.522428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44441 (local address 127.0.0.1:59082)
0926 07:24:56.523715 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:56.523729 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:56.523815 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:56.524422 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:56.524423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:56.524437 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:24:56.524514 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:56.524518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.525437 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.525443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:56.527123 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:56.527130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.528345 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.528349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:56.528545 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.528601 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 07:24:56.528678 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:24:56.529160 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"mutex_wait_us":476,"thread_start_us":77,"threads_started":1}
I20260926 07:24:56.529004 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.523334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.0.98.3:41533)
0926 07:24:56.523681 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:56.523693 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:56.523719 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:24:56.523791 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:56.523797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:56.523978 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:56.524142 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:24:56.524343 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.525482 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.526917 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.528208 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.528333 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.528465 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:56.528726 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 07:24:56.528814 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":100,"threads_started":1}
I20260926 07:24:56.532099 1357 tablet_service.cc:1467] Processing CreateTablet for tablet 8cff986360764f3389eda77cba7cdf2f (DEFAULT_TABLE table=test-table [id=f82a34e08e884b8684a1cf5f7ce0e000]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:24:56.532116 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 8cff986360764f3389eda77cba7cdf2f (DEFAULT_TABLE table=test-table [id=f82a34e08e884b8684a1cf5f7ce0e000]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:24:56.532119 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 8cff986360764f3389eda77cba7cdf2f (DEFAULT_TABLE table=test-table [id=f82a34e08e884b8684a1cf5f7ce0e000]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:24:56.533140 1357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cff986360764f3389eda77cba7cdf2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:56.533140 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cff986360764f3389eda77cba7cdf2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:56.533141 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cff986360764f3389eda77cba7cdf2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:56.539934 1442 tablet_bootstrap.cc:492] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22: Bootstrap starting.
I20260926 07:24:56.541108 1444 tablet_bootstrap.cc:492] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56: Bootstrap starting.
I20260926 07:24:56.542238 1442 tablet_bootstrap.cc:654] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:56.542814 1444 tablet_bootstrap.cc:654] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:56.543361 1443 tablet_bootstrap.cc:492] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090: Bootstrap starting.
I20260926 07:24:56.543514 1442 log.cc:824] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22: Log is configured to *not* fsync() on all Append() calls
I20260926 07:24:56.543653 1444 log.cc:824] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56: Log is configured to *not* fsync() on all Append() calls
I20260926 07:24:56.545567 1442 tablet_bootstrap.cc:492] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22: No bootstrap required, opened a new log
I20260926 07:24:56.545776 1442 ts_tablet_manager.cc:1397] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:24:56.545828 1443 tablet_bootstrap.cc:654] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:56.546586 1443 log.cc:824] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090: Log is configured to *not* fsync() on all Append() calls
I20260926 07:24:56.546774 1444 tablet_bootstrap.cc:492] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56: No bootstrap required, opened a new log
I20260926 07:24:56.546976 1444 ts_tablet_manager.cc:1397] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 07:24:56.548761 1443 tablet_bootstrap.cc:492] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090: No bootstrap required, opened a new log
I20260926 07:24:56.548934 1443 ts_tablet_manager.cc:1397] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:24:56.549588 1442 raft_consensus.cc:348] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20d3ac70fb046f8a1381f5156e64d56" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39225 } } peers { permanent_uuid: "56b785e3f90148758a1a7f5832d44090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } } peers { permanent_uuid: "38c43e829c5e4813ae1ed3344188ff22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41533 } }
I20260926 07:24:56.549924 1442 raft_consensus.cc:374] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:24:56.550036 1442 raft_consensus.cc:729] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c43e829c5e4813ae1ed3344188ff22, State: Initialized, Role: FOLLOWER
I20260926 07:24:56.550561 1442 consensus_queue.cc:260] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20d3ac70fb046f8a1381f5156e64d56" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39225 } } peers { permanent_uuid: "56b785e3f90148758a1a7f5832d44090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } } peers { permanent_uuid: "38c43e829c5e4813ae1ed3344188ff22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41533 } }
I20260926 07:24:56.551273 1423 heartbeater.cc:503] Master 127.0.98.62:45501 was elected leader, sending a full tablet report...
I20260926 07:24:56.551663 1442 ts_tablet_manager.cc:1428] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:24:56.551525 1444 raft_consensus.cc:348] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20d3ac70fb046f8a1381f5156e64d56" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39225 } } peers { permanent_uuid: "56b785e3f90148758a1a7f5832d44090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } } peers { permanent_uuid: "38c43e829c5e4813ae1ed3344188ff22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41533 } }
I20260926 07:24:56.552223 1444 raft_consensus.cc:374] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:24:56.552323 1444 raft_consensus.cc:729] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20d3ac70fb046f8a1381f5156e64d56, State: Initialized, Role: FOLLOWER
I20260926 07:24:56.552806 1444 consensus_queue.cc:260] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20d3ac70fb046f8a1381f5156e64d56" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39225 } } peers { permanent_uuid: "56b785e3f90148758a1a7f5832d44090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } } peers { permanent_uuid: "38c43e829c5e4813ae1ed3344188ff22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41533 } }
I20260926 07:24:56.553838 1444 ts_tablet_manager.cc:1428] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:24:56.554479 1443 raft_consensus.cc:348] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20d3ac70fb046f8a1381f5156e64d56" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39225 } } peers { permanent_uuid: "56b785e3f90148758a1a7f5832d44090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } } peers { permanent_uuid: "38c43e829c5e4813ae1ed3344188ff22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41533 } }
I20260926 07:24:56.554734 1443 raft_consensus.cc:374] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:24:56.554791 1443 raft_consensus.cc:729] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56b785e3f90148758a1a7f5832d44090, State: Initialized, Role: FOLLOWER
I20260926 07:24:56.555250 1443 consensus_queue.cc:260] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20d3ac70fb046f8a1381f5156e64d56" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39225 } } peers { permanent_uuid: "56b785e3f90148758a1a7f5832d44090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } } peers { permanent_uuid: "38c43e829c5e4813ae1ed3344188ff22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41533 } }
I20260926 07:24:56.556231 1443 ts_tablet_manager.cc:1428] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:24:56.556586 1288 heartbeater.cc:503] Master 127.0.98.62:45501 was elected leader, sending a full tablet report...
I20260926 07:24:56.622356 1449 raft_consensus.cc:482] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:24:56.622622 1449 raft_consensus.cc:504] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20d3ac70fb046f8a1381f5156e64d56" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39225 } } peers { permanent_uuid: "56b785e3f90148758a1a7f5832d44090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } } peers { permanent_uuid: "38c43e829c5e4813ae1ed3344188ff22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41533 } }
I20260926 07:24:56.624928 1449 leader_election.cc:288] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56b785e3f90148758a1a7f5832d44090 (127.0.98.2:44441), 38c43e829c5e4813ae1ed3344188ff22 (127.0.98.3:41533)
W20260926 07:24:56.629019 1289 tablet.cc:2369] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:24:56.633489 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.625352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51955 (local address 127.0.98.2:44441)
0926 07:24:56.625556 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:56.625560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:56.626887 (+ 1327us) server_negotiation.cc:407] Connection header received
0926 07:24:56.627151 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:56.627157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:56.627360 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:56.627472 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:24:56.629798 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.630543 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.632172 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.632816 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.632893 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.632960 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:56.633270 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 07:24:56.633358 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 07:24:56.633839 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.625153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44441 (local address 127.0.98.1:51955)
0926 07:24:56.626781 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:56.626797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:56.626985 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:56.629515 (+ 2530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:56.629518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:56.629539 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:24:56.629660 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:56.629666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.630685 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.630688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:56.632026 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:56.632036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.633052 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.633055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:56.633149 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.633171 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:24:56.633622 (+ 451us) client_negotiation.cc:240] Negotiation successful
0926 07:24:56.633698 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1533,"thread_start_us":743,"threads_started":1}
I20260926 07:24:56.634683 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.626266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40549 (local address 127.0.98.3:41533)
0926 07:24:56.626413 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:56.626418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:56.628841 (+ 2423us) server_negotiation.cc:407] Connection header received
0926 07:24:56.628914 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:56.628920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:56.628983 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:56.629095 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:24:56.629647 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.630695 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.632565 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.633435 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.633538 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.633605 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:56.634217 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 07:24:56.634532 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:24:56.635246 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.626192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41533 (local address 127.0.98.1:40549)
0926 07:24:56.628663 (+ 2471us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:56.628677 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:56.628782 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:56.629258 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:56.629264 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:56.629289 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:24:56.629468 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:56.629480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.630862 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.630865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:56.632382 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:56.632395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.633767 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.633771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:56.633895 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.634082 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 07:24:56.634895 (+ 813us) client_negotiation.cc:240] Negotiation successful
0926 07:24:56.634973 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2399,"thread_start_us":133,"threads_started":1}
I20260926 07:24:56.636097 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cff986360764f3389eda77cba7cdf2f" candidate_uuid: "c20d3ac70fb046f8a1381f5156e64d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56b785e3f90148758a1a7f5832d44090" is_pre_election: true
I20260926 07:24:56.636406 1242 raft_consensus.cc:2454] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c20d3ac70fb046f8a1381f5156e64d56 in term 0.
I20260926 07:24:56.636941 1041 leader_election.cc:302] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56b785e3f90148758a1a7f5832d44090, c20d3ac70fb046f8a1381f5156e64d56; no voters:
I20260926 07:24:56.637028 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cff986360764f3389eda77cba7cdf2f" candidate_uuid: "c20d3ac70fb046f8a1381f5156e64d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c43e829c5e4813ae1ed3344188ff22" is_pre_election: true
I20260926 07:24:56.637454 1377 raft_consensus.cc:2454] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c20d3ac70fb046f8a1381f5156e64d56 in term 0.
I20260926 07:24:56.638125 1449 raft_consensus.cc:2781] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:24:56.638222 1449 raft_consensus.cc:482] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:24:56.638284 1449 raft_consensus.cc:3037] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:56.643808 1449 raft_consensus.cc:504] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20d3ac70fb046f8a1381f5156e64d56" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39225 } } peers { permanent_uuid: "56b785e3f90148758a1a7f5832d44090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } } peers { permanent_uuid: "38c43e829c5e4813ae1ed3344188ff22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41533 } }
I20260926 07:24:56.644368 1449 leader_election.cc:288] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [CANDIDATE]: Term 1 election: Requested vote from peers 56b785e3f90148758a1a7f5832d44090 (127.0.98.2:44441), 38c43e829c5e4813ae1ed3344188ff22 (127.0.98.3:41533)
I20260926 07:24:56.644997 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cff986360764f3389eda77cba7cdf2f" candidate_uuid: "c20d3ac70fb046f8a1381f5156e64d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c43e829c5e4813ae1ed3344188ff22"
I20260926 07:24:56.645161 1377 raft_consensus.cc:3037] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:56.647073 1377 raft_consensus.cc:2454] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c20d3ac70fb046f8a1381f5156e64d56 in term 1.
I20260926 07:24:56.647565 1043 leader_election.cc:302] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c43e829c5e4813ae1ed3344188ff22, c20d3ac70fb046f8a1381f5156e64d56; no voters:
I20260926 07:24:56.648192 1449 raft_consensus.cc:2781] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:24:56.648250 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cff986360764f3389eda77cba7cdf2f" candidate_uuid: "c20d3ac70fb046f8a1381f5156e64d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56b785e3f90148758a1a7f5832d44090"
I20260926 07:24:56.648433 1242 raft_consensus.cc:3037] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:56.648469 1449 raft_consensus.cc:686] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [term 1 LEADER]: Becoming Leader. State: Replica: c20d3ac70fb046f8a1381f5156e64d56, State: Running, Role: LEADER
I20260926 07:24:56.648929 1449 consensus_queue.cc:237] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20d3ac70fb046f8a1381f5156e64d56" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39225 } } peers { permanent_uuid: "56b785e3f90148758a1a7f5832d44090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } } peers { permanent_uuid: "38c43e829c5e4813ae1ed3344188ff22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41533 } }
I20260926 07:24:56.649638 1242 raft_consensus.cc:2454] T 8cff986360764f3389eda77cba7cdf2f P 56b785e3f90148758a1a7f5832d44090 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c20d3ac70fb046f8a1381f5156e64d56 in term 1.
I20260926 07:24:56.652844 962 catalog_manager.cc:5495] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56 reported cstate change: term changed from 0 to 1, leader changed from <none> to c20d3ac70fb046f8a1381f5156e64d56 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c20d3ac70fb046f8a1381f5156e64d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20d3ac70fb046f8a1381f5156e64d56" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56b785e3f90148758a1a7f5832d44090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38c43e829c5e4813ae1ed3344188ff22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41533 } health_report { overall_health: UNKNOWN } } }
Sep 26 07:24:56 dist-test-slave-g04g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 07:24:56.685597 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.676305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40044 (local address 127.0.98.1:39225)
0926 07:24:56.676499 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:56.676503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:56.676626 (+ 123us) server_negotiation.cc:407] Connection header received
0926 07:24:56.676714 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:56.676717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:56.676765 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:56.676852 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:56.677511 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.678197 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.679020 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.679363 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.679511 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.681934 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:56.681946 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:56.681996 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:56.682033 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:56.683932 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:56.684337 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:56.684343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:56.684344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:56.684427 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:56.684662 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:56.684665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:56.684666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:56.684915 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:56.685078 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:56.685243 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 07:24:56.685378 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
Sep 26 07:24:56 dist-test-slave-g04g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407496, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260926 07:24:56.717963 1154 tablet.cc:2369] T 8cff986360764f3389eda77cba7cdf2f P c20d3ac70fb046f8a1381f5156e64d56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:24:56.726228 1424 tablet.cc:2369] T 8cff986360764f3389eda77cba7cdf2f P 38c43e829c5e4813ae1ed3344188ff22: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 07:24:56 dist-test-slave-g04g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407496, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 07:24:56.746763 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.737000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.0.98.1:39225)
0926 07:24:56.737136 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:56.737140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:56.737431 (+ 291us) server_negotiation.cc:407] Connection header received
0926 07:24:56.737530 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:56.737537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:56.737612 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:56.737721 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:56.738402 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.739411 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.740233 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.740732 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.740866 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.743483 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:56.743486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:56.743488 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:56.743511 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:56.744926 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:56.745367 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:56.745370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:56.745371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:56.745441 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:56.745733 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:56.745735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:56.745736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:56.745937 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:56.746039 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:56.746440 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 07:24:56.746547 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 07:24:56 dist-test-slave-g04g krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:24:56 dist-test-slave-g04g krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 07:24:56.841354 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:56.832462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.0.98.1:39225)
0926 07:24:56.832606 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:56.832611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:56.832807 (+ 196us) server_negotiation.cc:407] Connection header received
0926 07:24:56.832997 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:56.833001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:56.833044 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:56.833136 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:56.833899 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.834496 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.835408 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:56.835693 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:56.835803 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:56.838341 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:56.838344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:56.838346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:56.838368 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:56.839769 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:56.840175 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:56.840180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:56.840182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:56.840262 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:56.840534 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:56.840537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:56.840539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:56.840700 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:56.840833 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:56.841066 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 07:24:56.841184 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 07:24:56.842770 1067 tablet_service.cc:2908] Rejecting scan request for tablet 8cff986360764f3389eda77cba7cdf2f: Uninitialized: clean time has not yet been initialized
I20260926 07:24:56.844725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 1024
I20260926 07:24:56.851188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 1159
I20260926 07:24:56.856992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 1294
I20260926 07:24:56.861972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 930
2026-09-26T07:24:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5034 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:24:57 dist-test-slave-g04g krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:24:57 dist-test-slave-g04g krb5kdc[1467](info): set up 2 sockets
Sep 26 07:24:57 dist-test-slave-g04g krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260926 07:24:58.954674 392 mini_kdc.cc:121] Time spent starting KDC: real 2.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:24:59 dist-test-slave-g04g krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:24:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:24:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:24:59.076457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36603
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:59.183514 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:59.183785 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:59.183867 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:59.187980 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:24:59.188095 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:24:59.188133 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:59.188175 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:59.188216 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:24:59.188253 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:24:59.193468 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36603
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:59.194519 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:59.195575 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:59.204049 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:59.204071 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:59.204094 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:59.204196 1480 server_base.cc:1034] running on GCE node
I20260926 07:24:59.204880 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:59.206069 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:59.207290 1480 hybrid_clock.cc:648] HybridClock initialized: now 1790407499207256 us; error 39 us; skew 500 ppm
Sep 26 07:24:59 dist-test-slave-g04g krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:59.215247 1480 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:24:59.216151 1480 webserver.cc:460] Webserver started at http://127.0.98.62:39363/ using document root <none> and password file <none>
I20260926 07:24:59.216754 1480 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:59.216816 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:59.216969 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:59.218976 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "a48f80c6c3a445b5b10b01f1e8464276"
format_stamp: "Formatted at 2026-09-26 07:24:59 on dist-test-slave-g04g"
server_key: "762030a6ad823534393dd1f2efa242ab"
server_key_iv: "c1ba308aee7b325c62b668a266db5c1d"
server_key_version: "encryptionkey@0"
I20260926 07:24:59.219606 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "a48f80c6c3a445b5b10b01f1e8464276"
format_stamp: "Formatted at 2026-09-26 07:24:59 on dist-test-slave-g04g"
server_key: "762030a6ad823534393dd1f2efa242ab"
server_key_iv: "c1ba308aee7b325c62b668a266db5c1d"
server_key_version: "encryptionkey@0"
I20260926 07:24:59.223104 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:24:59.225358 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:59.226517 1480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:24:59.226610 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "a48f80c6c3a445b5b10b01f1e8464276"
format_stamp: "Formatted at 2026-09-26 07:24:59 on dist-test-slave-g04g"
server_key: "762030a6ad823534393dd1f2efa242ab"
server_key_iv: "c1ba308aee7b325c62b668a266db5c1d"
server_key_version: "encryptionkey@0"
I20260926 07:24:59.226686 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:59.237067 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:59.239790 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:59.239936 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:59.248143 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36603
I20260926 07:24:59.248167 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36603 every 8 connection(s)
I20260926 07:24:59.249075 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:24:59.252193 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:24:59.252285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 1480
I20260926 07:24:59.252389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:24:59.252743 392 external_mini_cluster.cc:1428] Setting key 5c0a1a8c87a81f1e1317fbd8c5886881
I20260926 07:24:59.260272 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276: Bootstrap starting.
I20260926 07:24:59.263586 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276: Neither blocks nor log segments found. Creating new log.
I20260926 07:24:59.264583 1558 log.cc:824] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:24:59 dist-test-slave-g04g krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:24:59.267308 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276: No bootstrap required, opened a new log
I20260926 07:24:59.272362 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f80c6c3a445b5b10b01f1e8464276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36603 } }
I20260926 07:24:59.272711 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:24:59.272807 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a48f80c6c3a445b5b10b01f1e8464276, State: Initialized, Role: FOLLOWER
I20260926 07:24:59.273434 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f80c6c3a445b5b10b01f1e8464276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36603 } }
I20260926 07:24:59.273689 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:24:59.273783 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:24:59.273910 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:24:59.275416 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f80c6c3a445b5b10b01f1e8464276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36603 } }
I20260926 07:24:59.275864 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:59.254314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.0.98.62:36603)
0926 07:24:59.254752 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:59.254767 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:59.254802 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:24:59.255794 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:59.255814 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:59.256203 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:59.256586 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:59.258188 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.260209 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.262255 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.263151 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.263526 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:59.267610 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:59.267618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:59.267638 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:59.267680 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:59.271622 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:59.272191 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:59.272197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:59.272203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:59.272334 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:59.272646 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:59.272651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:59.272653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:59.273507 (+ 854us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:59.273718 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:59.274205 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 07:24:59.274559 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":145,"threads_started":1}
I20260926 07:24:59.275866 1558 leader_election.cc:302] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a48f80c6c3a445b5b10b01f1e8464276; no voters:
I20260926 07:24:59.276290 1558 leader_election.cc:288] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:24:59.276765 1564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:24:59.277022 1564 raft_consensus.cc:686] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [term 1 LEADER]: Becoming Leader. State: Replica: a48f80c6c3a445b5b10b01f1e8464276, State: Running, Role: LEADER
I20260926 07:24:59.277356 1564 consensus_queue.cc:237] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f80c6c3a445b5b10b01f1e8464276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36603 } }
I20260926 07:24:59.277697 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:24:59.279055 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a48f80c6c3a445b5b10b01f1e8464276. Latest consensus state: current_term: 1 leader_uuid: "a48f80c6c3a445b5b10b01f1e8464276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f80c6c3a445b5b10b01f1e8464276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36603 } } }
I20260926 07:24:59.279210 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:59.279523 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a48f80c6c3a445b5b10b01f1e8464276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f80c6c3a445b5b10b01f1e8464276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36603 } } }
I20260926 07:24:59.279644 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276 [sys.catalog]: This master's current role is: LEADER
I20260926 07:24:59.279752 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:24:59.283222 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:24:59.289860 1572 catalog_manager.cc:1347] Generated new cluster ID: 08d0c7c9a12f43e19587d4c79d6406f0
I20260926 07:24:59.289933 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:24:59.299566 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:24:59.300534 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:24:59.320662 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276: Generated new TSK 0
I20260926 07:24:59.321599 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:24:59.360422 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36603
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:59.494771 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:59.495005 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:59.495113 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:59.499066 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:59.499116 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:59.499153 1584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:24:59.499184 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:24:59.504161 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:59.505069 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:59.505980 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:59.515344 1589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:59.515419 1592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:59.515441 1590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:59.515801 1584 server_base.cc:1034] running on GCE node
I20260926 07:24:59.516185 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:59.516785 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:59.517978 1584 hybrid_clock.cc:648] HybridClock initialized: now 1790407499517964 us; error 30 us; skew 500 ppm
Sep 26 07:24:59 dist-test-slave-g04g krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:59.523895 1584 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:24:59.524802 1584 webserver.cc:460] Webserver started at http://127.0.98.1:33221/ using document root <none> and password file <none>
I20260926 07:24:59.525372 1584 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:59.525462 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:59.525688 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:59.527771 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "7163d384e88646149c9832c0b002bb2a"
format_stamp: "Formatted at 2026-09-26 07:24:59 on dist-test-slave-g04g"
server_key: "aeb0e41abddd4c35307a7025c8b0cffb"
server_key_iv: "8f20707e3d2893b5614adab6839a6ba7"
server_key_version: "encryptionkey@0"
I20260926 07:24:59.528337 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7163d384e88646149c9832c0b002bb2a"
format_stamp: "Formatted at 2026-09-26 07:24:59 on dist-test-slave-g04g"
server_key: "aeb0e41abddd4c35307a7025c8b0cffb"
server_key_iv: "8f20707e3d2893b5614adab6839a6ba7"
server_key_version: "encryptionkey@0"
I20260926 07:24:59.532196 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:24:59.534475 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:59.535692 1584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:24:59.535825 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "7163d384e88646149c9832c0b002bb2a"
format_stamp: "Formatted at 2026-09-26 07:24:59 on dist-test-slave-g04g"
server_key: "aeb0e41abddd4c35307a7025c8b0cffb"
server_key_iv: "8f20707e3d2893b5614adab6839a6ba7"
server_key_version: "encryptionkey@0"
I20260926 07:24:59.535929 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:59.548022 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:59.550962 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:59.551189 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:59.552501 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:24:59.552574 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:59.552644 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:24:59.552695 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:24:59 dist-test-slave-g04g krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:24:59.563589 1584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41809
I20260926 07:24:59.563637 1714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41809 every 8 connection(s)
I20260926 07:24:59.564432 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:24:59.566682 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:59.553347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44525 (local address 127.0.98.62:36603)
0926 07:24:59.553473 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:59.553477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:59.554449 (+ 972us) server_negotiation.cc:407] Connection header received
0926 07:24:59.555502 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:59.555506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:59.555566 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:59.555647 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:59.557247 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.558159 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.559736 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.560159 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.560291 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:59.562981 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:59.562985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:59.562988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:59.563014 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:59.564770 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:59.565368 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:59.565372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:59.565374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:59.565480 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:59.565781 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:59.565782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:59.565784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:59.565946 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:59.566023 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:59.566468 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 07:24:59.566569 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:24:59.567263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 1584
I20260926 07:24:59.567375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:24:59.567660 392 external_mini_cluster.cc:1428] Setting key 849ace3097f7661f1a505a0fe29ae5d1
I20260926 07:24:59.567651 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:59.553864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36603 (local address 127.0.98.1:44525)
0926 07:24:59.554307 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:59.554345 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:59.555292 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:59.555959 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:59.555973 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:59.556476 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:59.557071 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:59.557089 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.558301 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.558304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:59.559603 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:59.559612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.560338 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.560344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:59.560529 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:59.561127 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:59.561142 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:59.562814 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:59.564883 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:59.564890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:59.564906 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:59.565262 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:59.565569 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:59.565571 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:59.565573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:59.565693 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:59.566053 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:59.566058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:59.566267 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 07:24:59.566411 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 07:24:59.566640 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":132,"threads_started":1}
I20260926 07:24:59.569074 1715 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36603
I20260926 07:24:59.569199 1715 heartbeater.cc:464] Registering TS with master...
I20260926 07:24:59.570021 1715 heartbeater.cc:511] Master 127.0.98.62:36603 requested a full tablet report, sending...
I20260926 07:24:59.571676 1511 ts_manager.cc:194] Registered new tserver with Master: 7163d384e88646149c9832c0b002bb2a (127.0.98.1:41809)
I20260926 07:24:59.573084 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44525
I20260926 07:24:59.579684 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:59.571501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42091 (local address 127.0.98.62:36603)
0926 07:24:59.571610 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:59.571614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:59.571839 (+ 225us) server_negotiation.cc:407] Connection header received
0926 07:24:59.572020 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:59.572023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:59.572065 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:59.572148 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:59.572716 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.573413 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.574666 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.574978 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.575160 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:59.576426 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:59.576429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:59.576432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:59.576455 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:59.577899 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:59.578418 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:59.578423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:59.578425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:59.578508 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:59.578833 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:59.578835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:59.578837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:59.578986 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:59.579132 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:59.579415 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 07:24:59.579547 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:24:59.579872 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:59.571357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36603 (local address 127.0.98.1:42091)
0926 07:24:59.571765 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:59.571779 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:59.571909 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:59.572186 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:59.572193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:59.572428 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:59.572589 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:59.572599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.573551 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.573558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:59.574527 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:59.574541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.575276 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.575280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:59.575370 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:59.575912 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:59.575928 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:59.576285 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:59.578017 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:59.578021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:59.578023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:59.578282 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:59.578636 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:59.578640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:59.578643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:59.578740 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:59.579132 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:59.579135 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:59.579249 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 07:24:59.579381 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 07:24:59.579562 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:24:59.597564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36603
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:59.705041 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:59.705277 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:59.705358 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:59.709357 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:59.709407 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:59.709479 1724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:24:59.709571 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:24:59.714939 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:59.715934 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:59.716883 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:59.726182 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:59.726202 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:59.726239 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:59.726323 1724 server_base.cc:1034] running on GCE node
I20260926 07:24:59.726881 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:59.727555 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:59.728717 1724 hybrid_clock.cc:648] HybridClock initialized: now 1790407499728690 us; error 40 us; skew 500 ppm
Sep 26 07:24:59 dist-test-slave-g04g krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:59.731496 1724 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:24:59.732340 1724 webserver.cc:460] Webserver started at http://127.0.98.2:40281/ using document root <none> and password file <none>
I20260926 07:24:59.732860 1724 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:59.732916 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:59.733134 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:59.735088 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "b0d482e99a2a4207bc7dd84eca11b552"
format_stamp: "Formatted at 2026-09-26 07:24:59 on dist-test-slave-g04g"
server_key: "4e2a92880345ed62e10d04cee345b74a"
server_key_iv: "2f78a9eeba223597f938d5285ef9c21e"
server_key_version: "encryptionkey@0"
I20260926 07:24:59.735642 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b0d482e99a2a4207bc7dd84eca11b552"
format_stamp: "Formatted at 2026-09-26 07:24:59 on dist-test-slave-g04g"
server_key: "4e2a92880345ed62e10d04cee345b74a"
server_key_iv: "2f78a9eeba223597f938d5285ef9c21e"
server_key_version: "encryptionkey@0"
I20260926 07:24:59.739171 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 07:24:59.741659 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:59.742901 1724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:24:59.743013 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "b0d482e99a2a4207bc7dd84eca11b552"
format_stamp: "Formatted at 2026-09-26 07:24:59 on dist-test-slave-g04g"
server_key: "4e2a92880345ed62e10d04cee345b74a"
server_key_iv: "2f78a9eeba223597f938d5285ef9c21e"
server_key_version: "encryptionkey@0"
I20260926 07:24:59.743148 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:59.763449 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:59.766566 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:59.766757 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:59.768136 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:24:59.768184 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:59.768246 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:24:59.768278 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:24:59 dist-test-slave-g04g krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:24:59.779093 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35533
I20260926 07:24:59.779304 1854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35533 every 8 connection(s)
I20260926 07:24:59.780045 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:24:59.782285 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:59.769028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53091 (local address 127.0.98.62:36603)
0926 07:24:59.769167 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:59.769172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:59.770368 (+ 1196us) server_negotiation.cc:407] Connection header received
0926 07:24:59.771533 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:59.771540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:59.771606 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:59.771712 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:59.772945 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.773874 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.775393 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.775626 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.775747 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:59.778231 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:59.778234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:59.778236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:59.778261 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:59.780194 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:59.780793 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:59.780799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:59.780801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:59.780896 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:59.781225 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:59.781228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:59.781230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:59.781417 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:59.781529 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:59.782000 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 07:24:59.782122 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:24:59.782964 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:59.769737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36603 (local address 127.0.98.2:53091)
0926 07:24:59.770217 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:59.770256 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:59.771299 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:59.771890 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:59.771897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:59.772334 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:59.772797 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:59.772816 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.774018 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.774023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:59.775264 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:59.775273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.775786 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.775789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:59.775964 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:59.776477 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:59.776492 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:59.778059 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:59.780305 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:59.780311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:59.780324 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:59.780663 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:59.780985 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:59.780988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:59.780990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:59.781118 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:59.781534 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:59.781543 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:59.781760 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 07:24:59.781933 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 07:24:59.782224 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":151,"threads_started":1}
I20260926 07:24:59.783157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 1724
I20260926 07:24:59.783273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:24:59.783550 392 external_mini_cluster.cc:1428] Setting key 6400b8a2296fc748cb272ee4c96f9d60
I20260926 07:24:59.784291 1855 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36603
I20260926 07:24:59.784416 1855 heartbeater.cc:464] Registering TS with master...
I20260926 07:24:59.784962 1855 heartbeater.cc:511] Master 127.0.98.62:36603 requested a full tablet report, sending...
I20260926 07:24:59.786850 1506 ts_manager.cc:194] Registered new tserver with Master: b0d482e99a2a4207bc7dd84eca11b552 (127.0.98.2:35533)
I20260926 07:24:59.787668 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53091
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 07:24:59.795177 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:59.787405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46791 (local address 127.0.98.62:36603)
0926 07:24:59.787977 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:59.787981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:59.788196 (+ 215us) server_negotiation.cc:407] Connection header received
0926 07:24:59.788357 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:59.788360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:59.788399 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:59.788505 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:59.788992 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.789452 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.790286 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.790682 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.790806 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:59.791881 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:59.791885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:59.791887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:59.791913 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:24:59.793546 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:59.793961 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:59.793967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:59.793968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:59.794054 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:24:59.794302 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:24:59.794305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:24:59.794307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:24:59.794605 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:24:59.794715 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:24:59.794907 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 07:24:59.794992 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482}
I20260926 07:24:59.795202 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:59.787329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36603 (local address 127.0.98.2:46791)
0926 07:24:59.788120 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:59.788134 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:59.788253 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:59.788520 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:59.788523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:59.788708 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:59.788796 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:59.788801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.789588 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.789591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:59.790172 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:59.790180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.790797 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.790799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:59.790872 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:59.791442 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:59.791452 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:59.791743 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:24:59.793659 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:59.793662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:59.793663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:59.793855 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:59.794145 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:24:59.794147 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:24:59.794148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:24:59.794219 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:24:59.794701 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:24:59.794703 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:24:59.794774 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 07:24:59.794856 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 07:24:59.794993 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:24:59.813531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36603
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:24:59.921161 1864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:24:59.921388 1864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:24:59.921429 1864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:24:59.925595 1864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:24:59.925644 1864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:24:59.925707 1864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:24:59.925786 1864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:24:59.930796 1864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:24:59.931787 1864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:24:59.932768 1864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:24:59.942693 1872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:59.942708 1870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:24:59.942793 1869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:24:59.942744 1864 server_base.cc:1034] running on GCE node
I20260926 07:24:59.943506 1864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:24:59.944197 1864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:24:59.945415 1864 hybrid_clock.cc:648] HybridClock initialized: now 1790407499945396 us; error 36 us; skew 500 ppm
Sep 26 07:24:59 dist-test-slave-g04g krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:24:59.948151 1864 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:24:59.949054 1864 webserver.cc:460] Webserver started at http://127.0.98.3:46329/ using document root <none> and password file <none>
I20260926 07:24:59.949659 1864 fs_manager.cc:359] Metadata directory not provided
I20260926 07:24:59.949720 1864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:24:59.949945 1864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:24:59.952037 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "b4391965b45548ffbd04d23ac8217ab8"
format_stamp: "Formatted at 2026-09-26 07:24:59 on dist-test-slave-g04g"
server_key: "36371d505f9df341e21b36956ffabe63"
server_key_iv: "bf03e3048b78dcf11ed0d9bdd019572a"
server_key_version: "encryptionkey@0"
I20260926 07:24:59.952615 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b4391965b45548ffbd04d23ac8217ab8"
format_stamp: "Formatted at 2026-09-26 07:24:59 on dist-test-slave-g04g"
server_key: "36371d505f9df341e21b36956ffabe63"
server_key_iv: "bf03e3048b78dcf11ed0d9bdd019572a"
server_key_version: "encryptionkey@0"
I20260926 07:24:59.956502 1864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:24:59.958923 1879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:59.960141 1864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:24:59.960269 1864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "b4391965b45548ffbd04d23ac8217ab8"
format_stamp: "Formatted at 2026-09-26 07:24:59 on dist-test-slave-g04g"
server_key: "36371d505f9df341e21b36956ffabe63"
server_key_iv: "bf03e3048b78dcf11ed0d9bdd019572a"
server_key_version: "encryptionkey@0"
I20260926 07:24:59.960377 1864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:24:59.983394 1864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:24:59.986500 1864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:24:59.986699 1864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:24:59.988107 1864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:24:59.988180 1864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:24:59.988250 1864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:24:59.988302 1864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:24:59 dist-test-slave-g04g krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:24:59.999207 1864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45373
I20260926 07:24:59.999496 1994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45373 every 8 connection(s)
I20260926 07:25:00.000334 1864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:00.002519 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:59.988933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55043 (local address 127.0.98.62:36603)
0926 07:24:59.989092 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 07:24:59.989096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:24:59.990173 (+ 1077us) server_negotiation.cc:407] Connection header received
0926 07:24:59.991313 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:24:59.991321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:24:59.991391 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:24:59.991493 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:24:59.993028 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.993583 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.994832 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.995145 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.995275 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:59.998246 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:24:59.998250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:24:59.998259 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:24:59.998284 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:00.000196 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:00.000799 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:00.000804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:00.000805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:00.000903 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:00.001273 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:00.001276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:00.001278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:00.001468 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:00.001551 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:00.002168 (+ 617us) server_negotiation.cc:299] Negotiation successful
0926 07:25:00.002313 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 07:25:00.003345 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:24:59.989534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36603 (local address 127.0.98.3:55043)
0926 07:24:59.990030 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 07:24:59.990070 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:24:59.991080 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:24:59.991707 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:24:59.991713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:24:59.992197 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:24:59.992877 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:59.992893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.993731 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.993735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:59.994706 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:24:59.994715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:24:59.995407 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:24:59.995416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:24:59.995558 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:24:59.996168 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:24:59.996183 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:24:59.998050 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:00.000316 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:00.000322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:00.000335 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:00.000678 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:00.001011 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:00.001014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:00.001016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:00.001178 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:00.001583 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:00.001592 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:00.001880 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 07:25:00.002076 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 07:25:00.002392 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":140,"threads_started":1}
I20260926 07:25:00.004491 1995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36603
I20260926 07:25:00.004618 1995 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:00.005151 1995 heartbeater.cc:511] Master 127.0.98.62:36603 requested a full tablet report, sending...
I20260926 07:25:00.006201 1511 ts_manager.cc:194] Registered new tserver with Master: b4391965b45548ffbd04d23ac8217ab8 (127.0.98.3:45373)
I20260926 07:25:00.007017 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55043
I20260926 07:25:00.009428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 1864
I20260926 07:25:00.009567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:00.009872 392 external_mini_cluster.cc:1428] Setting key 1c1d377a75b7d96bc8311cbf45d09449
I20260926 07:25:00.011822 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:25:00.014333 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.006710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47491 (local address 127.0.98.62:36603)
0926 07:25:00.006815 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:00.006819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:00.007026 (+ 207us) server_negotiation.cc:407] Connection header received
0926 07:25:00.007165 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:00.007168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:00.007213 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:00.007315 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:00.007786 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.008513 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.009284 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.009633 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.009750 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.010868 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:00.010875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:00.010878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:00.010905 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:00.012696 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:00.013090 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:00.013096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:00.013100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:00.013188 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:00.013499 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:00.013505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:00.013509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:00.013740 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:00.013853 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:00.014144 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 07:25:00.014230 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:25:00.014505 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.006625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36603 (local address 127.0.98.3:47491)
0926 07:25:00.006911 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:00.006960 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:00.007078 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:00.007350 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:00.007353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:00.007568 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:00.007682 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.007687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.008636 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.008638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.009177 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.009184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.009745 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.009748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.009816 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.010371 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:00.010382 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:00.010752 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:00.012807 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:00.012810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:00.012811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:00.012998 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:00.013312 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:00.013315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:00.013316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:00.013402 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:00.013862 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:00.013866 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:00.013972 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 07:25:00.014093 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 07:25:00.014254 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
Sep 26 07:25:00 dist-test-slave-g04g krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:25:00 dist-test-slave-g04g krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:00.038414 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.030589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.0.98.62:36603)
0926 07:25:00.030689 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:00.030692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:00.030702 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:25:00.030748 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:00.030750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:00.030783 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:00.030871 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:00.031590 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.032100 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.032925 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.033141 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.033249 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.035593 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:00.035597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:00.035599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:00.035629 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:00.036825 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:00.037241 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:00.037246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:00.037248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:00.037327 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:00.037598 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:00.037603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:00.037605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:00.037768 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:00.037907 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:00.038144 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 07:25:00.038252 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:25:00.041311 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:25:00.044306 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:00.058584 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.053387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49188 (local address 127.0.98.1:41809)
0926 07:25:00.053772 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:00.053787 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:00.053896 (+ 109us) server_negotiation.cc:407] Connection header received
0926 07:25:00.054037 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:00.054042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:00.054220 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:00.054368 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:00.054604 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.055443 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.057403 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.057932 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.058056 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.058225 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:00.058349 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 07:25:00.058439 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":97,"threads_started":1}
I20260926 07:25:00.058575 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.052976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41809 (local address 127.0.0.1:49188)
0926 07:25:00.053814 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:00.053826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:00.053908 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:00.054356 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:00.054359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:00.054376 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:00.054491 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.054496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.055561 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.055564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.057276 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.057282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.058053 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.058057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.058166 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.058202 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:25:00.058315 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 07:25:00.058428 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":196,"thread_start_us":83,"threads_started":1}
I20260926 07:25:00.058954 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.052523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45373 (local address 127.0.0.1:37534)
0926 07:25:00.053430 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:00.053478 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:00.053610 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:00.054142 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:00.054147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:00.054168 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:00.054282 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.054289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.055707 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.055710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.057263 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.057274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.058392 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.058395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.058580 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.058654 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 07:25:00.058733 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:25:00.058807 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":227,"spinlock_wait_cycles":4480,"thread_start_us":88,"threads_started":1}
I20260926 07:25:00.059190 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.052449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35533 (local address 127.0.0.1:60264)
0926 07:25:00.053431 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:00.053483 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:00.053612 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:00.054520 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:00.054523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:00.054537 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:00.054667 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.054673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.055995 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.055999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.057102 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.057111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.058387 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.058391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.058578 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.058792 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 07:25:00.058851 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 07:25:00.058906 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":127,"spinlock_wait_cycles":4736,"thread_start_us":67,"threads_started":1}
I20260926 07:25:00.059648 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.052923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.0.98.3:45373)
0926 07:25:00.053643 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:00.053657 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:00.053688 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:25:00.053764 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:00.053769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:00.053980 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:00.054142 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:00.054405 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.055599 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.057383 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.058137 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.058223 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.058345 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:00.059406 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0926 07:25:00.059509 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":117,"threads_started":1}
I20260926 07:25:00.059648 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.052552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.0.98.2:35533)
0926 07:25:00.054021 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:00.054032 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:00.054050 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:25:00.054109 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:00.054113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:00.054247 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:00.054368 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:00.054791 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.055853 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.057223 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.058046 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.058152 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.058320 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:00.059423 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0926 07:25:00.059509 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1351,"thread_start_us":98,"threads_started":1}
I20260926 07:25:00.061615 1649 tablet_service.cc:1467] Processing CreateTablet for tablet c16cc24994db4913bd97e481aa388624 (DEFAULT_TABLE table=test-table [id=40a5d9bc55b74eb892acef5dba852375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:00.062275 1789 tablet_service.cc:1467] Processing CreateTablet for tablet c16cc24994db4913bd97e481aa388624 (DEFAULT_TABLE table=test-table [id=40a5d9bc55b74eb892acef5dba852375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:00.062731 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16cc24994db4913bd97e481aa388624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:00.063294 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16cc24994db4913bd97e481aa388624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:00.064522 1929 tablet_service.cc:1467] Processing CreateTablet for tablet c16cc24994db4913bd97e481aa388624 (DEFAULT_TABLE table=test-table [id=40a5d9bc55b74eb892acef5dba852375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:00.065330 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16cc24994db4913bd97e481aa388624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:00.070720 2018 tablet_bootstrap.cc:492] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a: Bootstrap starting.
I20260926 07:25:00.071936 2019 tablet_bootstrap.cc:492] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552: Bootstrap starting.
I20260926 07:25:00.072829 2018 tablet_bootstrap.cc:654] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:00.073556 2018 log.cc:824] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:00.073630 2019 tablet_bootstrap.cc:654] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:00.074273 2020 tablet_bootstrap.cc:492] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8: Bootstrap starting.
I20260926 07:25:00.074457 2019 log.cc:824] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:00.075599 2018 tablet_bootstrap.cc:492] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a: No bootstrap required, opened a new log
I20260926 07:25:00.075788 2018 ts_tablet_manager.cc:1397] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:25:00.076143 2019 tablet_bootstrap.cc:492] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552: No bootstrap required, opened a new log
I20260926 07:25:00.076198 2020 tablet_bootstrap.cc:654] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:00.076330 2019 ts_tablet_manager.cc:1397] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 07:25:00.076941 2020 log.cc:824] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:00.078881 2020 tablet_bootstrap.cc:492] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8: No bootstrap required, opened a new log
I20260926 07:25:00.079111 2020 ts_tablet_manager.cc:1397] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 07:25:00.079669 2018 raft_consensus.cc:348] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } }
I20260926 07:25:00.079723 2019 raft_consensus.cc:348] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } }
I20260926 07:25:00.079932 2018 raft_consensus.cc:374] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:00.079958 2019 raft_consensus.cc:374] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:00.079988 2018 raft_consensus.cc:729] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7163d384e88646149c9832c0b002bb2a, State: Initialized, Role: FOLLOWER
I20260926 07:25:00.080008 2019 raft_consensus.cc:729] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d482e99a2a4207bc7dd84eca11b552, State: Initialized, Role: FOLLOWER
I20260926 07:25:00.080473 2018 consensus_queue.cc:260] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } }
I20260926 07:25:00.080473 2019 consensus_queue.cc:260] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } }
I20260926 07:25:00.081385 2019 ts_tablet_manager.cc:1428] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 07:25:00.081403 2018 ts_tablet_manager.cc:1428] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:25:00.081754 1715 heartbeater.cc:503] Master 127.0.98.62:36603 was elected leader, sending a full tablet report...
I20260926 07:25:00.082647 1855 heartbeater.cc:503] Master 127.0.98.62:36603 was elected leader, sending a full tablet report...
I20260926 07:25:00.082832 2020 raft_consensus.cc:348] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } }
I20260926 07:25:00.083057 2020 raft_consensus.cc:374] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:00.083112 2020 raft_consensus.cc:729] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4391965b45548ffbd04d23ac8217ab8, State: Initialized, Role: FOLLOWER
I20260926 07:25:00.083544 2020 consensus_queue.cc:260] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } }
I20260926 07:25:00.084610 1995 heartbeater.cc:503] Master 127.0.98.62:36603 was elected leader, sending a full tablet report...
I20260926 07:25:00.084820 2020 ts_tablet_manager.cc:1428] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:25:00.155875 2026 raft_consensus.cc:482] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:00.156068 2026 raft_consensus.cc:504] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } }
I20260926 07:25:00.157222 2026 leader_election.cc:288] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0d482e99a2a4207bc7dd84eca11b552 (127.0.98.2:35533), 7163d384e88646149c9832c0b002bb2a (127.0.98.1:41809)
I20260926 07:25:00.160964 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.157538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41809 (local address 127.0.98.3:55057)
0926 07:25:00.157830 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:00.157843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:00.157928 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:00.158161 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:00.158164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:00.158182 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:00.158327 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.158332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.159060 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.159065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.159903 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.159914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.160521 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.160524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.160623 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.160647 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:25:00.160742 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 07:25:00.160809 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":23,"thread_start_us":110,"threads_started":1}
I20260926 07:25:00.160945 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.157620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55057 (local address 127.0.98.1:41809)
0926 07:25:00.157743 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:00.157748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:00.157928 (+ 180us) server_negotiation.cc:407] Connection header received
0926 07:25:00.158006 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:00.158009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:00.158052 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:00.158148 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:00.158446 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.158933 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.160026 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.160399 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.160521 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.160592 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:00.160755 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 07:25:00.160820 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:25:00.161273 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.157436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35533 (local address 127.0.98.3:35247)
0926 07:25:00.157555 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:00.157573 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:00.157664 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:00.158055 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:00.158058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:00.158070 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:00.158208 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.158213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.158947 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.158953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.160195 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.160204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.160846 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.160850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.160946 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.160971 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:25:00.161057 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 07:25:00.161146 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":2}
I20260926 07:25:00.161430 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.157661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35247 (local address 127.0.98.2:35533)
0926 07:25:00.157826 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:00.157831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:00.157846 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:25:00.157905 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:00.157910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:00.157976 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:00.158050 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:00.158326 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.158836 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.160323 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.160722 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.160836 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.160893 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:00.161229 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 07:25:00.161313 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 07:25:00.161541 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16cc24994db4913bd97e481aa388624" candidate_uuid: "b4391965b45548ffbd04d23ac8217ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7163d384e88646149c9832c0b002bb2a" is_pre_election: true
I20260926 07:25:00.161856 1668 raft_consensus.cc:2454] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4391965b45548ffbd04d23ac8217ab8 in term 0.
I20260926 07:25:00.161891 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16cc24994db4913bd97e481aa388624" candidate_uuid: "b4391965b45548ffbd04d23ac8217ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d482e99a2a4207bc7dd84eca11b552" is_pre_election: true
I20260926 07:25:00.162242 1809 raft_consensus.cc:2454] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4391965b45548ffbd04d23ac8217ab8 in term 0.
I20260926 07:25:00.162357 1883 leader_election.cc:302] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7163d384e88646149c9832c0b002bb2a, b4391965b45548ffbd04d23ac8217ab8; no voters:
I20260926 07:25:00.162685 2026 raft_consensus.cc:2781] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:00.162782 2026 raft_consensus.cc:482] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:00.162825 2026 raft_consensus.cc:3037] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:00.163919 2026 raft_consensus.cc:504] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } }
I20260926 07:25:00.164310 2026 leader_election.cc:288] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [CANDIDATE]: Term 1 election: Requested vote from peers b0d482e99a2a4207bc7dd84eca11b552 (127.0.98.2:35533), 7163d384e88646149c9832c0b002bb2a (127.0.98.1:41809)
I20260926 07:25:00.164606 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16cc24994db4913bd97e481aa388624" candidate_uuid: "b4391965b45548ffbd04d23ac8217ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d482e99a2a4207bc7dd84eca11b552"
I20260926 07:25:00.164753 1809 raft_consensus.cc:3037] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:00.164749 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16cc24994db4913bd97e481aa388624" candidate_uuid: "b4391965b45548ffbd04d23ac8217ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7163d384e88646149c9832c0b002bb2a"
I20260926 07:25:00.164904 1668 raft_consensus.cc:3037] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:00.165784 1809 raft_consensus.cc:2454] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4391965b45548ffbd04d23ac8217ab8 in term 1.
I20260926 07:25:00.165979 1668 raft_consensus.cc:2454] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4391965b45548ffbd04d23ac8217ab8 in term 1.
I20260926 07:25:00.166100 1880 leader_election.cc:302] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0d482e99a2a4207bc7dd84eca11b552, b4391965b45548ffbd04d23ac8217ab8; no voters:
I20260926 07:25:00.166278 2026 raft_consensus.cc:2781] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:00.166502 2026 raft_consensus.cc:686] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [term 1 LEADER]: Becoming Leader. State: Replica: b4391965b45548ffbd04d23ac8217ab8, State: Running, Role: LEADER
I20260926 07:25:00.166834 2026 consensus_queue.cc:237] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } }
I20260926 07:25:00.170066 1512 catalog_manager.cc:5495] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4391965b45548ffbd04d23ac8217ab8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b4391965b45548ffbd04d23ac8217ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } health_report { overall_health: UNKNOWN } } }
I20260926 07:25:00.214329 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.210980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.0.98.3:45373)
0926 07:25:00.211131 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:00.211135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:00.211149 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:00.211315 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:00.211317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:00.211371 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:00.211471 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:00.211710 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.212192 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.213025 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.213239 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.213314 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.213606 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:00.213988 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:00.214070 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:00.214176 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 07:25:00.214223 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:25:00.220116 1668 raft_consensus.cc:1261] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a [term 1 FOLLOWER]: Refusing update from remote peer b4391965b45548ffbd04d23ac8217ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:00.220413 1809 raft_consensus.cc:1261] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552 [term 1 FOLLOWER]: Refusing update from remote peer b4391965b45548ffbd04d23ac8217ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:00.220811 2028 consensus_queue.cc:1035] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:00.221048 2026 consensus_queue.cc:1035] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:00.228889 2032 mvcc.cc:205] Tried to move back new op lower bound from 7333509120893894656 to 7333509120686137344. Current Snapshot: MvccSnapshot[applied={T|T < 7333509120893894656}]
I20260926 07:25:00.229272 2033 mvcc.cc:205] Tried to move back new op lower bound from 7333509120893894656 to 7333509120686137344. Current Snapshot: MvccSnapshot[applied={T|T < 7333509120893894656}]
I20260926 07:25:00.238163 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:00.238328 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:00.240695 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276: Sending DeleteTablet for 3 replicas of tablet c16cc24994db4913bd97e481aa388624
I20260926 07:25:00.241406 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet c16cc24994db4913bd97e481aa388624 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37534
I20260926 07:25:00.241480 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet c16cc24994db4913bd97e481aa388624 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60264
I20260926 07:25:00.241851 2040 tablet_replica.cc:330] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8: stopping tablet replica
I20260926 07:25:00.242192 2040 raft_consensus.cc:2229] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:25:00.242159 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet c16cc24994db4913bd97e481aa388624 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49188
I20260926 07:25:00.242264 2041 tablet_replica.cc:330] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552: stopping tablet replica
I20260926 07:25:00.242533 2041 raft_consensus.cc:2229] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:00.242822 2041 raft_consensus.cc:2258] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:00.242865 2040 raft_consensus.cc:2258] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:00.243201 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:25:00.243611 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:00.244643 2041 ts_tablet_manager.cc:1905] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:00.244669 2040 ts_tablet_manager.cc:1905] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:00.247464 2041 ts_tablet_manager.cc:1918] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:00.247560 2041 log.cc:1196] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/c16cc24994db4913bd97e481aa388624
I20260926 07:25:00.247517 2040 ts_tablet_manager.cc:1918] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:00.247982 2041 ts_tablet_manager.cc:1939] T c16cc24994db4913bd97e481aa388624 P b0d482e99a2a4207bc7dd84eca11b552: Deleting consensus metadata
I20260926 07:25:00.248714 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 86935e65cad6414184574b2ad80dc8e9 (DEFAULT_TABLE table=test-table [id=5b898864da2440e9b676b831321b2107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:00.248730 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 86935e65cad6414184574b2ad80dc8e9 (DEFAULT_TABLE table=test-table [id=5b898864da2440e9b676b831321b2107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:00.248962 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86935e65cad6414184574b2ad80dc8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:00.248981 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86935e65cad6414184574b2ad80dc8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:00.251219 2018 tablet_bootstrap.cc:492] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a: Bootstrap starting.
I20260926 07:25:00.251286 2019 tablet_bootstrap.cc:492] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552: Bootstrap starting.
I20260926 07:25:00.251926 1496 catalog_manager.cc:4841] TS b0d482e99a2a4207bc7dd84eca11b552 (127.0.98.2:35533): tablet c16cc24994db4913bd97e481aa388624 (table test-table [id=40a5d9bc55b74eb892acef5dba852375]) successfully deleted
I20260926 07:25:00.252159 2019 tablet_bootstrap.cc:654] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:00.252230 2018 tablet_bootstrap.cc:654] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:00.253255 2019 tablet_bootstrap.cc:492] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552: No bootstrap required, opened a new log
I20260926 07:25:00.253320 2019 ts_tablet_manager.cc:1397] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:00.253714 2019 raft_consensus.cc:348] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.253813 2019 raft_consensus.cc:374] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:00.253841 2019 raft_consensus.cc:729] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d482e99a2a4207bc7dd84eca11b552, State: Initialized, Role: FOLLOWER
I20260926 07:25:00.253964 2019 consensus_queue.cc:260] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.254191 2019 ts_tablet_manager.cc:1428] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:00.254280 2042 tablet_replica.cc:330] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a: stopping tablet replica
I20260926 07:25:00.254547 2042 raft_consensus.cc:2229] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:00.254798 2042 raft_consensus.cc:2258] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:00.257004 2042 ts_tablet_manager.cc:1905] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:00.247773 2040 log.cc:1196] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/c16cc24994db4913bd97e481aa388624
I20260926 07:25:00.253013 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 86935e65cad6414184574b2ad80dc8e9 (DEFAULT_TABLE table=test-table [id=5b898864da2440e9b676b831321b2107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:00.259325 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86935e65cad6414184574b2ad80dc8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:00.259557 2042 ts_tablet_manager.cc:1918] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:00.259650 2042 log.cc:1196] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/wal/wals/c16cc24994db4913bd97e481aa388624
I20260926 07:25:00.260356 2042 ts_tablet_manager.cc:1939] T c16cc24994db4913bd97e481aa388624 P 7163d384e88646149c9832c0b002bb2a: Deleting consensus metadata
I20260926 07:25:00.261592 1499 catalog_manager.cc:4841] TS 7163d384e88646149c9832c0b002bb2a (127.0.98.1:41809): tablet c16cc24994db4913bd97e481aa388624 (table test-table [id=40a5d9bc55b74eb892acef5dba852375]) successfully deleted
I20260926 07:25:00.261754 2020 tablet_bootstrap.cc:492] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8: Bootstrap starting.
I20260926 07:25:00.262719 2020 tablet_bootstrap.cc:654] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:00.263796 2040 ts_tablet_manager.cc:1939] T c16cc24994db4913bd97e481aa388624 P b4391965b45548ffbd04d23ac8217ab8: Deleting consensus metadata
I20260926 07:25:00.263844 2018 tablet_bootstrap.cc:492] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a: No bootstrap required, opened a new log
I20260926 07:25:00.263942 2018 ts_tablet_manager.cc:1397] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260926 07:25:00.264396 2018 raft_consensus.cc:348] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.264518 2018 raft_consensus.cc:374] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:00.264554 2018 raft_consensus.cc:729] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7163d384e88646149c9832c0b002bb2a, State: Initialized, Role: FOLLOWER
I20260926 07:25:00.264684 2018 consensus_queue.cc:260] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.265022 2018 ts_tablet_manager.cc:1428] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:00.265319 1498 catalog_manager.cc:4841] TS b4391965b45548ffbd04d23ac8217ab8 (127.0.98.3:45373): tablet c16cc24994db4913bd97e481aa388624 (table test-table [id=40a5d9bc55b74eb892acef5dba852375]) successfully deleted
I20260926 07:25:00.267879 2020 tablet_bootstrap.cc:492] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8: No bootstrap required, opened a new log
I20260926 07:25:00.267980 2020 ts_tablet_manager.cc:1397] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260926 07:25:00.268362 2020 raft_consensus.cc:348] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.268514 2020 raft_consensus.cc:374] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:00.268558 2020 raft_consensus.cc:729] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4391965b45548ffbd04d23ac8217ab8, State: Initialized, Role: FOLLOWER
I20260926 07:25:00.268695 2020 consensus_queue.cc:260] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.269109 2020 ts_tablet_manager.cc:1428] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260926 07:25:00.281204 1856 tablet.cc:2369] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:25:00.315984 1716 tablet.cc:2369] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:25:00.503026 1996 tablet.cc:2369] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:25:00.648691 2028 raft_consensus.cc:482] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:00.648880 2028 raft_consensus.cc:504] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.649313 2028 leader_election.cc:288] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7163d384e88646149c9832c0b002bb2a (127.0.98.1:41809), b0d482e99a2a4207bc7dd84eca11b552 (127.0.98.2:35533)
I20260926 07:25:00.649802 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86935e65cad6414184574b2ad80dc8e9" candidate_uuid: "b4391965b45548ffbd04d23ac8217ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7163d384e88646149c9832c0b002bb2a" is_pre_election: true
I20260926 07:25:00.649816 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86935e65cad6414184574b2ad80dc8e9" candidate_uuid: "b4391965b45548ffbd04d23ac8217ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d482e99a2a4207bc7dd84eca11b552" is_pre_election: true
I20260926 07:25:00.649963 1668 raft_consensus.cc:2454] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4391965b45548ffbd04d23ac8217ab8 in term 0.
I20260926 07:25:00.649988 1809 raft_consensus.cc:2454] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4391965b45548ffbd04d23ac8217ab8 in term 0.
I20260926 07:25:00.650281 1883 leader_election.cc:302] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7163d384e88646149c9832c0b002bb2a, b4391965b45548ffbd04d23ac8217ab8; no voters:
I20260926 07:25:00.650470 2028 raft_consensus.cc:2781] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:00.650530 2028 raft_consensus.cc:482] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:00.650607 2028 raft_consensus.cc:3037] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:00.651697 2028 raft_consensus.cc:504] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.652073 2028 leader_election.cc:288] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [CANDIDATE]: Term 1 election: Requested vote from peers 7163d384e88646149c9832c0b002bb2a (127.0.98.1:41809), b0d482e99a2a4207bc7dd84eca11b552 (127.0.98.2:35533)
I20260926 07:25:00.652385 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86935e65cad6414184574b2ad80dc8e9" candidate_uuid: "b4391965b45548ffbd04d23ac8217ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7163d384e88646149c9832c0b002bb2a"
I20260926 07:25:00.652455 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86935e65cad6414184574b2ad80dc8e9" candidate_uuid: "b4391965b45548ffbd04d23ac8217ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d482e99a2a4207bc7dd84eca11b552"
I20260926 07:25:00.652518 1668 raft_consensus.cc:3037] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:00.652599 1809 raft_consensus.cc:3037] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:00.653409 1668 raft_consensus.cc:2454] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4391965b45548ffbd04d23ac8217ab8 in term 1.
I20260926 07:25:00.653566 1809 raft_consensus.cc:2454] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4391965b45548ffbd04d23ac8217ab8 in term 1.
I20260926 07:25:00.653712 1883 leader_election.cc:302] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7163d384e88646149c9832c0b002bb2a, b4391965b45548ffbd04d23ac8217ab8; no voters:
I20260926 07:25:00.653892 2028 raft_consensus.cc:2781] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:00.654002 2028 raft_consensus.cc:686] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [term 1 LEADER]: Becoming Leader. State: Replica: b4391965b45548ffbd04d23ac8217ab8, State: Running, Role: LEADER
I20260926 07:25:00.654143 2028 consensus_queue.cc:237] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.655772 1511 catalog_manager.cc:5495] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4391965b45548ffbd04d23ac8217ab8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b4391965b45548ffbd04d23ac8217ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } health_report { overall_health: HEALTHY } } }
Sep 26 07:25:00 dist-test-slave-g04g krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:00.749203 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.740697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36603 (local address 127.0.0.1:56264)
0926 07:25:00.741042 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:00.741058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:00.741212 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:00.741493 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:00.741496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:00.741917 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:00.742007 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.742014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.742865 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.742871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.743560 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:00.743571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.744024 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.744028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:00.744100 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.744635 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:00.744646 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:00.746027 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:00.747761 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:00.747767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:00.747773 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:00.747995 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:00.748233 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:00.748235 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:00.748236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:00.748309 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:00.748664 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:00.748670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:00.748780 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 07:25:00.748893 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 07:25:00.749024 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":101,"threads_started":1}
I20260926 07:25:00.749209 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:00.740784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.0.98.62:36603)
0926 07:25:00.741022 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:00.741032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:00.741146 (+ 114us) server_negotiation.cc:407] Connection header received
0926 07:25:00.741316 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:00.741319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:00.741366 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:00.741468 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:00.742137 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.742673 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.743679 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:00.743916 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:00.744025 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:00.746145 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:00.746148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:00.746151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:00.746171 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:00.747666 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:00.748078 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:00.748081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:00.748082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:00.748155 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:00.748395 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:00.748397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:00.748398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:00.748548 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:00.748626 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:00.748960 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 07:25:00.749080 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260926 07:25:00.752827 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56264:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:25:00.753597 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:00.758440 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 8251427fc4bd40e0a709127d531313cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2042915ff4634934a2464599ab52f885]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:25:00.758503 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 8251427fc4bd40e0a709127d531313cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2042915ff4634934a2464599ab52f885]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:25:00.758836 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8251427fc4bd40e0a709127d531313cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:00.758836 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8251427fc4bd40e0a709127d531313cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:00.758731 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 8251427fc4bd40e0a709127d531313cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2042915ff4634934a2464599ab52f885]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:25:00.759152 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8251427fc4bd40e0a709127d531313cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:00.761431 2020 tablet_bootstrap.cc:492] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8: Bootstrap starting.
I20260926 07:25:00.761739 2018 tablet_bootstrap.cc:492] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a: Bootstrap starting.
I20260926 07:25:00.762065 2054 tablet_bootstrap.cc:492] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552: Bootstrap starting.
I20260926 07:25:00.762413 2020 tablet_bootstrap.cc:654] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:00.762571 2018 tablet_bootstrap.cc:654] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:00.762773 2054 tablet_bootstrap.cc:654] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:00.763602 2020 tablet_bootstrap.cc:492] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8: No bootstrap required, opened a new log
I20260926 07:25:00.763691 2020 ts_tablet_manager.cc:1397] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:00.763840 2018 tablet_bootstrap.cc:492] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a: No bootstrap required, opened a new log
I20260926 07:25:00.763942 2018 ts_tablet_manager.cc:1397] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 07:25:00.764129 2020 raft_consensus.cc:348] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.764242 2020 raft_consensus.cc:374] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:00.764286 2020 raft_consensus.cc:729] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4391965b45548ffbd04d23ac8217ab8, State: Initialized, Role: FOLLOWER
I20260926 07:25:00.764362 2018 raft_consensus.cc:348] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.764405 2020 consensus_queue.cc:260] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.764474 2018 raft_consensus.cc:374] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:00.764537 2018 raft_consensus.cc:729] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7163d384e88646149c9832c0b002bb2a, State: Initialized, Role: FOLLOWER
I20260926 07:25:00.764671 2020 ts_tablet_manager.cc:1428] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:00.764658 2018 consensus_queue.cc:260] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.764905 2018 ts_tablet_manager.cc:1428] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:25:00.764905 2028 tablet_replica.cc:439] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } }
I20260926 07:25:00.765030 2028 tablet_replica.cc:442] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:00.765120 2024 tablet_replica.cc:439] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } }
I20260926 07:25:00.765249 2024 tablet_replica.cc:442] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:00.766812 2054 tablet_bootstrap.cc:492] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552: No bootstrap required, opened a new log
I20260926 07:25:00.766911 2054 ts_tablet_manager.cc:1397] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260926 07:25:00.767385 2054 raft_consensus.cc:348] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.767524 2054 raft_consensus.cc:374] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:00.767568 2054 raft_consensus.cc:729] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d482e99a2a4207bc7dd84eca11b552, State: Initialized, Role: FOLLOWER
I20260926 07:25:00.767707 2054 consensus_queue.cc:260] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:00.768069 2054 ts_tablet_manager.cc:1428] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:00.768222 2058 tablet_replica.cc:439] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } }
I20260926 07:25:00.768406 2058 tablet_replica.cc:442] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:01.114948 2028 raft_consensus.cc:482] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:01.115127 2028 raft_consensus.cc:504] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:01.115617 2028 leader_election.cc:288] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7163d384e88646149c9832c0b002bb2a (127.0.98.1:41809), b0d482e99a2a4207bc7dd84eca11b552 (127.0.98.2:35533)
I20260926 07:25:01.116086 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8251427fc4bd40e0a709127d531313cb" candidate_uuid: "b4391965b45548ffbd04d23ac8217ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7163d384e88646149c9832c0b002bb2a" is_pre_election: true
I20260926 07:25:01.116185 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8251427fc4bd40e0a709127d531313cb" candidate_uuid: "b4391965b45548ffbd04d23ac8217ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d482e99a2a4207bc7dd84eca11b552" is_pre_election: true
I20260926 07:25:01.116271 1668 raft_consensus.cc:2454] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4391965b45548ffbd04d23ac8217ab8 in term 0.
I20260926 07:25:01.116391 1809 raft_consensus.cc:2454] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4391965b45548ffbd04d23ac8217ab8 in term 0.
I20260926 07:25:01.116678 1883 leader_election.cc:302] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7163d384e88646149c9832c0b002bb2a, b4391965b45548ffbd04d23ac8217ab8; no voters:
I20260926 07:25:01.116870 2028 raft_consensus.cc:2781] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:01.116945 2028 raft_consensus.cc:482] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:01.116976 2028 raft_consensus.cc:3037] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:01.118024 2028 raft_consensus.cc:504] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:01.118404 2028 leader_election.cc:288] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [CANDIDATE]: Term 1 election: Requested vote from peers 7163d384e88646149c9832c0b002bb2a (127.0.98.1:41809), b0d482e99a2a4207bc7dd84eca11b552 (127.0.98.2:35533)
I20260926 07:25:01.118805 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8251427fc4bd40e0a709127d531313cb" candidate_uuid: "b4391965b45548ffbd04d23ac8217ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d482e99a2a4207bc7dd84eca11b552"
I20260926 07:25:01.118805 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8251427fc4bd40e0a709127d531313cb" candidate_uuid: "b4391965b45548ffbd04d23ac8217ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7163d384e88646149c9832c0b002bb2a"
I20260926 07:25:01.118937 1809 raft_consensus.cc:3037] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:01.118955 1668 raft_consensus.cc:3037] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:01.120081 1809 raft_consensus.cc:2454] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4391965b45548ffbd04d23ac8217ab8 in term 1.
I20260926 07:25:01.120081 1668 raft_consensus.cc:2454] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4391965b45548ffbd04d23ac8217ab8 in term 1.
I20260926 07:25:01.120412 1880 leader_election.cc:302] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0d482e99a2a4207bc7dd84eca11b552, b4391965b45548ffbd04d23ac8217ab8; no voters:
I20260926 07:25:01.120607 2028 raft_consensus.cc:2781] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:01.120733 2028 raft_consensus.cc:686] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [term 1 LEADER]: Becoming Leader. State: Replica: b4391965b45548ffbd04d23ac8217ab8, State: Running, Role: LEADER
I20260926 07:25:01.120869 2028 consensus_queue.cc:237] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } }
I20260926 07:25:01.121598 2026 tablet_replica.cc:439] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8: TxnStatusTablet state changed. Reason: New leader b4391965b45548ffbd04d23ac8217ab8. Latest consensus state: current_term: 1 leader_uuid: "b4391965b45548ffbd04d23ac8217ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } }
I20260926 07:25:01.121709 2026 tablet_replica.cc:442] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:25:01.121927 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:25:01.122087 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:25:01.122532 1506 catalog_manager.cc:5495] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4391965b45548ffbd04d23ac8217ab8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b4391965b45548ffbd04d23ac8217ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } health_report { overall_health: HEALTHY } } }
I20260926 07:25:01.140043 2026 consensus_queue.cc:1035] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:01.143565 2028 consensus_queue.cc:1035] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:01.219308 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:01.215209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.0.98.3:45373)
0926 07:25:01.215521 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:01.215527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:01.215540 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:25:01.215615 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:01.215619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:01.215671 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:01.215745 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:01.216007 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:01.216526 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:01.217754 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:01.218124 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:01.218680 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:01.218758 (+ 78us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:01.218839 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:01.218926 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:01.219125 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 07:25:01.219182 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":161,"threads_started":1}
I20260926 07:25:01.219326 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:01.215151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45373 (local address 127.0.0.1:37544)
0926 07:25:01.215326 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:01.215342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:01.215470 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:01.215782 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:01.215785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:01.215793 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:25:01.215900 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:01.215906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:01.216644 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:01.216648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:01.217587 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:01.217599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:01.218297 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:01.218304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:01.218434 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:01.218499 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:25:01.218949 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:25:01.218965 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 07:25:01.219084 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 07:25:01.219175 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 07:25:01.222110 1668 raft_consensus.cc:1261] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a [term 1 FOLLOWER]: Refusing update from remote peer b4391965b45548ffbd04d23ac8217ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:01.222132 1809 raft_consensus.cc:1261] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552 [term 1 FOLLOWER]: Refusing update from remote peer b4391965b45548ffbd04d23ac8217ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:01.222616 2028 consensus_queue.cc:1035] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:01.222775 2026 consensus_queue.cc:1035] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:01.224217 2058 tablet_replica.cc:439] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552: TxnStatusTablet state changed. Reason: New leader b4391965b45548ffbd04d23ac8217ab8. Latest consensus state: current_term: 1 leader_uuid: "b4391965b45548ffbd04d23ac8217ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } }
I20260926 07:25:01.224373 2058 tablet_replica.cc:442] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:01.224555 2024 tablet_replica.cc:439] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a: TxnStatusTablet state changed. Reason: New leader b4391965b45548ffbd04d23ac8217ab8. Latest consensus state: current_term: 1 leader_uuid: "b4391965b45548ffbd04d23ac8217ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } }
I20260926 07:25:01.224679 2024 tablet_replica.cc:442] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:01.227064 2026 tablet_replica.cc:439] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4391965b45548ffbd04d23ac8217ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } }
I20260926 07:25:01.227206 2026 tablet_replica.cc:442] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:25:01.227367 2026 tablet_replica.cc:439] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4391965b45548ffbd04d23ac8217ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } }
I20260926 07:25:01.227451 2026 tablet_replica.cc:442] T 8251427fc4bd40e0a709127d531313cb P b4391965b45548ffbd04d23ac8217ab8: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:25:01.227548 2063 mvcc.cc:205] Tried to move back new op lower bound from 7333509125001236480 to 7333509124592840704. Current Snapshot: MvccSnapshot[applied={T|T < 7333509125001236480 or (T in {7333509125001236480})}]
I20260926 07:25:01.228739 2024 tablet_replica.cc:439] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4391965b45548ffbd04d23ac8217ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } }
I20260926 07:25:01.228863 2024 tablet_replica.cc:442] T 8251427fc4bd40e0a709127d531313cb P 7163d384e88646149c9832c0b002bb2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:01.229771 2058 tablet_replica.cc:439] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4391965b45548ffbd04d23ac8217ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7163d384e88646149c9832c0b002bb2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41809 } } peers { permanent_uuid: "b0d482e99a2a4207bc7dd84eca11b552" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35533 } } peers { permanent_uuid: "b4391965b45548ffbd04d23ac8217ab8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45373 } } }
I20260926 07:25:01.229890 2058 tablet_replica.cc:442] T 8251427fc4bd40e0a709127d531313cb P b0d482e99a2a4207bc7dd84eca11b552: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:01.243541 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:01.240185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37255 (local address 127.0.98.3:45373)
0926 07:25:01.240327 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:01.240332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:01.240525 (+ 193us) server_negotiation.cc:407] Connection header received
0926 07:25:01.240675 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:01.240678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:01.240717 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:01.240800 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:01.241071 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:01.241583 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:01.242450 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:01.242669 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:01.242783 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:01.243077 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:01.243147 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:01.243225 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:01.243316 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 07:25:01.243386 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":60}
I20260926 07:25:01.243541 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:01.240105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45373 (local address 127.0.98.3:37255)
0926 07:25:01.240438 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:01.240451 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:01.240573 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:01.240844 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:01.240847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:01.240853 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:25:01.240973 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:01.240978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:01.241690 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:01.241696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:01.242315 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:01.242323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:01.242790 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:01.242793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:01.242893 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:01.242928 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:25:01.243226 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:25:01.243233 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:25:01.243308 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:25:01.243403 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":26,"thread_start_us":91,"threads_started":1}
I20260926 07:25:01.282817 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:01.283000 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:01.284921 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a48f80c6c3a445b5b10b01f1e8464276: Sending DeleteTablet for 3 replicas of tablet 86935e65cad6414184574b2ad80dc8e9
I20260926 07:25:01.285511 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 86935e65cad6414184574b2ad80dc8e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60264
I20260926 07:25:01.285550 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 86935e65cad6414184574b2ad80dc8e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37534
I20260926 07:25:01.285665 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet 86935e65cad6414184574b2ad80dc8e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49188
I20260926 07:25:01.285876 2080 tablet_replica.cc:330] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8: stopping tablet replica
I20260926 07:25:01.285926 2081 tablet_replica.cc:330] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a: stopping tablet replica
I20260926 07:25:01.286001 2080 raft_consensus.cc:2229] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:25:01.286026 2081 raft_consensus.cc:2229] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:01.286124 2081 raft_consensus.cc:2258] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:01.286188 2080 raft_consensus.cc:2258] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:01.286860 2079 tablet_replica.cc:330] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552: stopping tablet replica
I20260926 07:25:01.286893 2081 ts_tablet_manager.cc:1905] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:01.286990 2079 raft_consensus.cc:2229] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:01.286888 2080 ts_tablet_manager.cc:1905] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:01.287142 2079 raft_consensus.cc:2258] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:01.287719 2079 ts_tablet_manager.cc:1905] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:01.289734 2080 ts_tablet_manager.cc:1918] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:01.289733 2081 ts_tablet_manager.cc:1918] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:01.289814 2080 log.cc:1196] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/86935e65cad6414184574b2ad80dc8e9
I20260926 07:25:01.289814 2081 log.cc:1196] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-0/wal/wals/86935e65cad6414184574b2ad80dc8e9
I20260926 07:25:01.289916 2079 ts_tablet_manager.cc:1918] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:01.289997 2079 log.cc:1196] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/86935e65cad6414184574b2ad80dc8e9
I20260926 07:25:01.290081 2081 ts_tablet_manager.cc:1939] T 86935e65cad6414184574b2ad80dc8e9 P 7163d384e88646149c9832c0b002bb2a: Deleting consensus metadata
I20260926 07:25:01.290081 2080 ts_tablet_manager.cc:1939] T 86935e65cad6414184574b2ad80dc8e9 P b4391965b45548ffbd04d23ac8217ab8: Deleting consensus metadata
I20260926 07:25:01.290289 2079 ts_tablet_manager.cc:1939] T 86935e65cad6414184574b2ad80dc8e9 P b0d482e99a2a4207bc7dd84eca11b552: Deleting consensus metadata
I20260926 07:25:01.290979 1496 catalog_manager.cc:4841] TS b0d482e99a2a4207bc7dd84eca11b552 (127.0.98.2:35533): tablet 86935e65cad6414184574b2ad80dc8e9 (table test-table [id=5b898864da2440e9b676b831321b2107]) successfully deleted
I20260926 07:25:01.291059 1498 catalog_manager.cc:4841] TS b4391965b45548ffbd04d23ac8217ab8 (127.0.98.3:45373): tablet 86935e65cad6414184574b2ad80dc8e9 (table test-table [id=5b898864da2440e9b676b831321b2107]) successfully deleted
I20260926 07:25:01.291059 1499 catalog_manager.cc:4841] TS 7163d384e88646149c9832c0b002bb2a (127.0.98.1:41809): tablet 86935e65cad6414184574b2ad80dc8e9 (table test-table [id=5b898864da2440e9b676b831321b2107]) successfully deleted
Sep 26 07:25:01 dist-test-slave-g04g krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 07:25:01.300798 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:01.286979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.0.98.1:41809)
0926 07:25:01.290331 (+ 3352us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:01.290335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:01.291094 (+ 759us) server_negotiation.cc:407] Connection header received
0926 07:25:01.291284 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:01.291287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:01.291347 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:01.291440 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:01.292100 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:01.292627 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:01.293440 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:01.293688 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:01.293799 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:01.297058 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:01.297067 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:01.297081 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:01.297104 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:01.298745 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:01.299345 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:01.299352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:01.299353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:01.299438 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:01.299750 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:01.299754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:01.299756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:01.300027 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:01.300289 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:01.300446 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 07:25:01.300581 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3276,"thread_start_us":74,"threads_started":1}
W20260926 07:25:01.301299 1706 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49200
I20260926 07:25:01.311391 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:01.303493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.0.98.62:36603)
0926 07:25:01.303765 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:01.303770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:01.303784 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:01.303964 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:01.303968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:01.304046 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:01.304150 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:01.304845 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:01.305456 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:01.306321 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:01.306598 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:01.306721 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:01.308154 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:01.308158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:01.308161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:01.308197 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:01.309568 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:01.310027 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:01.310031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:01.310033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:01.310122 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:01.310424 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:01.310430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:01.310432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:01.310624 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:01.310778 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:01.311028 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 07:25:01.311192 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":66,"threads_started":1}
W20260926 07:25:01.311815 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56276
I20260926 07:25:01.313102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 1584
I20260926 07:25:01.319346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 1724
I20260926 07:25:01.325420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 1864
I20260926 07:25:01.332242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 1480
2026-09-26T07:25:01Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4468 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:25:01 dist-test-slave-g04g krb5kdc[2090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:25:01 dist-test-slave-g04g krb5kdc[2090](info): set up 2 sockets
Sep 26 07:25:01 dist-test-slave-g04g krb5kdc[2090](info): commencing operation
krb5kdc: starting...
W20260926 07:25:02.379194 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:25:02 dist-test-slave-g04g krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:25:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:25:02.501492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34689
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:02.608067 2103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:02.608337 2103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:02.608378 2103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:02.613014 2103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:02.613075 2103 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:25:02.613091 2103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:02.613109 2103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:02.613125 2103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:02.613142 2103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:02.618310 2103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34689
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:02.619311 2103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:02.620620 2103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:02.630883 2108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:02.630913 2109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:02.630928 2111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:02.631285 2103 server_base.cc:1034] running on GCE node
I20260926 07:25:02.631804 2103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:02.632863 2103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:02.634060 2103 hybrid_clock.cc:648] HybridClock initialized: now 1790407502634046 us; error 37 us; skew 500 ppm
Sep 26 07:25:02 dist-test-slave-g04g krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:02.636852 2103 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:25:02.637676 2103 webserver.cc:460] Webserver started at http://127.0.98.62:45863/ using document root <none> and password file <none>
I20260926 07:25:02.638230 2103 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:02.638289 2103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:02.638520 2103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:02.640533 2103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "ab426e4e6ebb4059a0c8852ffa2e92f3"
format_stamp: "Formatted at 2026-09-26 07:25:02 on dist-test-slave-g04g"
server_key: "bd395d88099ccd6b5bec26290d0dcf4c"
server_key_iv: "9affb60ad9eac617ed6868a74df41c4f"
server_key_version: "encryptionkey@0"
I20260926 07:25:02.641088 2103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab426e4e6ebb4059a0c8852ffa2e92f3"
format_stamp: "Formatted at 2026-09-26 07:25:02 on dist-test-slave-g04g"
server_key: "bd395d88099ccd6b5bec26290d0dcf4c"
server_key_iv: "9affb60ad9eac617ed6868a74df41c4f"
server_key_version: "encryptionkey@0"
I20260926 07:25:02.644645 2103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260926 07:25:02.646906 2118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:02.648099 2103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260926 07:25:02.648234 2103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "ab426e4e6ebb4059a0c8852ffa2e92f3"
format_stamp: "Formatted at 2026-09-26 07:25:02 on dist-test-slave-g04g"
server_key: "bd395d88099ccd6b5bec26290d0dcf4c"
server_key_iv: "9affb60ad9eac617ed6868a74df41c4f"
server_key_version: "encryptionkey@0"
I20260926 07:25:02.648331 2103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:02.667315 2103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:02.670586 2103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:02.670786 2103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:02.678612 2103 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34689
I20260926 07:25:02.678707 2180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34689 every 8 connection(s)
I20260926 07:25:02.679584 2103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:02.682281 2181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:02.687394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 2103
I20260926 07:25:02.687507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:02.687767 392 external_mini_cluster.cc:1428] Setting key 971377a223b6e74171c60c032727e566
I20260926 07:25:02.687829 2181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3: Bootstrap starting.
I20260926 07:25:02.690235 2181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:02.691066 2181 log.cc:824] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:02.692971 2181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3: No bootstrap required, opened a new log
Sep 26 07:25:02 dist-test-slave-g04g krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:02.697194 2181 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab426e4e6ebb4059a0c8852ffa2e92f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } }
I20260926 07:25:02.697459 2181 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:02.697513 2181 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab426e4e6ebb4059a0c8852ffa2e92f3, State: Initialized, Role: FOLLOWER
I20260926 07:25:02.697966 2181 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab426e4e6ebb4059a0c8852ffa2e92f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } }
I20260926 07:25:02.698124 2181 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:02.698194 2181 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:02.698372 2181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:02.699255 2181 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab426e4e6ebb4059a0c8852ffa2e92f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } }
I20260926 07:25:02.699576 2181 leader_election.cc:302] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab426e4e6ebb4059a0c8852ffa2e92f3; no voters:
I20260926 07:25:02.699904 2181 leader_election.cc:288] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:02.699977 2186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:02.700165 2186 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [term 1 LEADER]: Becoming Leader. State: Replica: ab426e4e6ebb4059a0c8852ffa2e92f3, State: Running, Role: LEADER
I20260926 07:25:02.700510 2186 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab426e4e6ebb4059a0c8852ffa2e92f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } }
I20260926 07:25:02.701172 2181 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:02.702291 2188 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab426e4e6ebb4059a0c8852ffa2e92f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab426e4e6ebb4059a0c8852ffa2e92f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } } }
I20260926 07:25:02.702409 2188 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:02.702430 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:02.689037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.0.98.62:34689)
0926 07:25:02.689435 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:02.689452 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:02.689488 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:25:02.690125 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:02.690143 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:02.690524 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:02.690801 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:02.691697 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:02.692708 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:02.693833 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:02.694281 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:02.694395 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:02.697084 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:02.697089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:02.697101 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:02.697126 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:02.699209 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:02.699675 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:02.699681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:02.699712 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:02.699851 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:02.700150 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:02.700154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:02.700156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:02.700697 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:02.700924 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:02.701180 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 07:25:02.701481 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":140,"threads_started":1}
I20260926 07:25:02.703392 2187 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab426e4e6ebb4059a0c8852ffa2e92f3. Latest consensus state: current_term: 1 leader_uuid: "ab426e4e6ebb4059a0c8852ffa2e92f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab426e4e6ebb4059a0c8852ffa2e92f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34689 } } }
I20260926 07:25:02.703490 2187 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3 [sys.catalog]: This master's current role is: LEADER
W20260926 07:25:02.706188 2202 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:25:02.706269 2202 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:25:02.706398 2195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:02.707201 2195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:02.712821 2195 catalog_manager.cc:1347] Generated new cluster ID: fbc1fe837cae4cb781b4ba19f4cdcdad
I20260926 07:25:02.712880 2195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:02.726517 2195 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:02.727569 2195 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:02.738408 2195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3: Generated new TSK 0
I20260926 07:25:02.739122 2195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:25:02.770277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34689
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:02.882360 2207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:02.882627 2207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:02.882717 2207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:02.886786 2207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:02.886845 2207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:02.886888 2207 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:25:02.886943 2207 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:25:02.887012 2207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:02.892282 2207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:02.893302 2207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:02.894327 2207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:02.903818 2213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:02.903901 2215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:02.903833 2212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:02.904137 2207 server_base.cc:1034] running on GCE node
I20260926 07:25:02.904552 2207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:02.905205 2207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:02.906407 2207 hybrid_clock.cc:648] HybridClock initialized: now 1790407502906392 us; error 38 us; skew 500 ppm
Sep 26 07:25:02 dist-test-slave-g04g krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:02.909242 2207 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:25:02.910724 2207 webserver.cc:460] Webserver started at http://127.0.98.1:35805/ using document root <none> and password file <none>
I20260926 07:25:02.911352 2207 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:02.911445 2207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:02.911687 2207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:02.913688 2207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "b4e41537987f45039495bb0e26570aea"
format_stamp: "Formatted at 2026-09-26 07:25:02 on dist-test-slave-g04g"
server_key: "19eed613ba9c700db67e235230466504"
server_key_iv: "1bbe3035345d8018e00bc8896515b290"
server_key_version: "encryptionkey@0"
I20260926 07:25:02.914273 2207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b4e41537987f45039495bb0e26570aea"
format_stamp: "Formatted at 2026-09-26 07:25:02 on dist-test-slave-g04g"
server_key: "19eed613ba9c700db67e235230466504"
server_key_iv: "1bbe3035345d8018e00bc8896515b290"
server_key_version: "encryptionkey@0"
I20260926 07:25:02.918069 2207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:25:02.920481 2222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:02.921715 2207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:02.921859 2207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "b4e41537987f45039495bb0e26570aea"
format_stamp: "Formatted at 2026-09-26 07:25:02 on dist-test-slave-g04g"
server_key: "19eed613ba9c700db67e235230466504"
server_key_iv: "1bbe3035345d8018e00bc8896515b290"
server_key_version: "encryptionkey@0"
I20260926 07:25:02.921967 2207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:02.932524 2207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:02.935194 2207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:02.935331 2207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:02.936720 2207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:02.936793 2207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:02.936864 2207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:02.936918 2207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:25:02 dist-test-slave-g04g krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:02.947672 2207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44567
I20260926 07:25:02.947703 2337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44567 every 8 connection(s)
I20260926 07:25:02.948758 2207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:02.950407 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:02.937307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53453 (local address 127.0.98.62:34689)
0926 07:25:02.937451 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:02.937456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:02.938704 (+ 1248us) server_negotiation.cc:407] Connection header received
0926 07:25:02.939800 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:02.939804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:02.939860 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:02.939938 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:02.941156 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:02.941770 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:02.943120 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:02.943419 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:02.943510 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:02.946609 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:02.946613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:02.946616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:02.946651 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:02.948302 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:02.948871 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:02.948875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:02.948877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:02.948963 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:02.949397 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:02.949402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:02.949407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:02.949646 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:02.949747 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:02.950185 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 07:25:02.950288 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:25:02.951299 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:02.938098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:53453)
0926 07:25:02.938539 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:02.938586 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:02.939596 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:02.940093 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:02.940100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:02.940410 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:02.941011 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:02.941025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:02.941893 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:02.941897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:02.942964 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:02.942975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:02.943681 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:02.943685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:02.943872 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:02.944528 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:02.944550 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:02.946417 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:02.948414 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:02.948420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:02.948435 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:02.948758 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:02.949066 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:02.949072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:02.949085 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:02.949262 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:02.949751 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:02.949757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:02.949970 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 07:25:02.950125 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 07:25:02.950390 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":141,"threads_started":1}
I20260926 07:25:02.952770 2338 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34689
I20260926 07:25:02.952961 2338 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:02.953517 2338 heartbeater.cc:511] Master 127.0.98.62:34689 requested a full tablet report, sending...
I20260926 07:25:02.955250 2134 ts_manager.cc:194] Registered new tserver with Master: b4e41537987f45039495bb0e26570aea (127.0.98.1:44567)
I20260926 07:25:02.956411 2134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53453
I20260926 07:25:02.956609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 2207
I20260926 07:25:02.956735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:25:02.957072 392 external_mini_cluster.cc:1428] Setting key 33c4fc3990b65a279c5409781a6c4f2e
I20260926 07:25:02.962628 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:02.954794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54931 (local address 127.0.98.62:34689)
0926 07:25:02.954931 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:02.954937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:02.955197 (+ 260us) server_negotiation.cc:407] Connection header received
0926 07:25:02.955312 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:02.955315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:02.955379 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:02.955450 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:02.955976 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:02.956569 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:02.957641 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:02.957953 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:02.958051 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:02.959389 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:02.959396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:02.959402 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:02.959432 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:02.960898 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:02.961409 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:02.961411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:02.961412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:02.961478 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:02.961805 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:02.961807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:02.961808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:02.961962 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:02.962036 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:02.962414 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 07:25:02.962513 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":47}
I20260926 07:25:02.962787 2344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:02.954774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.1:54931)
0926 07:25:02.955104 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:02.955133 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:02.955236 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:02.955553 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:02.955555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:02.955722 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:02.955815 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:02.955868 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:02.956725 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:02.956729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:02.957466 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:02.957478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:02.958088 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:02.958092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:02.958184 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:02.958745 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:02.958763 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:02.959255 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:02.961015 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:02.961021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:02.961027 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:02.961308 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:02.961579 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:02.961585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:02.961590 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:02.961703 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:02.962074 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:02.962080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:02.962199 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 07:25:02.962331 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 07:25:02.962501 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:25:02.985639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34689
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:03.092864 2347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:03.093106 2347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:03.093152 2347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:03.097023 2347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:03.097076 2347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:03.097114 2347 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:25:03.097129 2347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:25:03.097160 2347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:03.102257 2347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:03.103180 2347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:03.104095 2347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:03.113261 2352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:03.113327 2355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:03.113250 2353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:03.113534 2347 server_base.cc:1034] running on GCE node
I20260926 07:25:03.113950 2347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:03.114583 2347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:03.115737 2347 hybrid_clock.cc:648] HybridClock initialized: now 1790407503115724 us; error 25 us; skew 500 ppm
Sep 26 07:25:03 dist-test-slave-g04g krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:03.118557 2347 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:25:03.119494 2347 webserver.cc:460] Webserver started at http://127.0.98.2:36201/ using document root <none> and password file <none>
I20260926 07:25:03.120129 2347 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:03.120208 2347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:03.120437 2347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:03.122478 2347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "f82f3787270748cda591c8ba0476f589"
format_stamp: "Formatted at 2026-09-26 07:25:03 on dist-test-slave-g04g"
server_key: "4ba01b1e1d5202387041a4346b8a8394"
server_key_iv: "7fbaa41faff3bfae25d11422abb38842"
server_key_version: "encryptionkey@0"
I20260926 07:25:03.123148 2347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f82f3787270748cda591c8ba0476f589"
format_stamp: "Formatted at 2026-09-26 07:25:03 on dist-test-slave-g04g"
server_key: "4ba01b1e1d5202387041a4346b8a8394"
server_key_iv: "7fbaa41faff3bfae25d11422abb38842"
server_key_version: "encryptionkey@0"
I20260926 07:25:03.126842 2347 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:03.129181 2362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:03.130405 2347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:03.130518 2347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "f82f3787270748cda591c8ba0476f589"
format_stamp: "Formatted at 2026-09-26 07:25:03 on dist-test-slave-g04g"
server_key: "4ba01b1e1d5202387041a4346b8a8394"
server_key_iv: "7fbaa41faff3bfae25d11422abb38842"
server_key_version: "encryptionkey@0"
I20260926 07:25:03.130614 2347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:03.149969 2347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:03.153100 2347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:03.153288 2347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:03.154939 2347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:03.155033 2347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:03.155169 2347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:03.155231 2347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:25:03 dist-test-slave-g04g krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:03.166258 2347 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33295
I20260926 07:25:03.166316 2477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33295 every 8 connection(s)
I20260926 07:25:03.167349 2347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:25:03.168691 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.155559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53739 (local address 127.0.98.62:34689)
0926 07:25:03.155750 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:03.155754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:03.156600 (+ 846us) server_negotiation.cc:407] Connection header received
0926 07:25:03.157506 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:03.157510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:03.157563 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:03.157664 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:03.159455 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.160371 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.161710 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.162083 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.162221 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.164772 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:03.164779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:03.164785 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:03.164821 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:03.166587 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:03.167075 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:03.167079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:03.167081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:03.167169 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:03.167549 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:03.167552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:03.167554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:03.167843 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:03.167937 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:03.168474 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 07:25:03.168571 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 07:25:03.169492 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.155994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:53739)
0926 07:25:03.156440 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:03.156487 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:03.157311 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:03.158005 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:03.158015 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:03.158563 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:03.159303 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.159322 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.160534 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.160538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.161609 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.161618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.162265 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.162269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.162472 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.163104 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:03.163122 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:03.164593 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:03.166691 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:03.166695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:03.166706 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:03.166938 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:03.167270 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:03.167273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:03.167275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:03.167447 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:03.167985 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:03.167991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:03.168262 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 07:25:03.168413 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 07:25:03.168653 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":128,"threads_started":1}
I20260926 07:25:03.170888 2478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34689
I20260926 07:25:03.171041 2478 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:03.171277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 2347
I20260926 07:25:03.171396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:25:03.171617 2478 heartbeater.cc:511] Master 127.0.98.62:34689 requested a full tablet report, sending...
I20260926 07:25:03.171751 392 external_mini_cluster.cc:1428] Setting key 618a3134377828125a6b8e1e41a0a9be
I20260926 07:25:03.172607 2134 ts_manager.cc:194] Registered new tserver with Master: f82f3787270748cda591c8ba0476f589 (127.0.98.2:33295)
I20260926 07:25:03.173381 2134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53739
I20260926 07:25:03.181061 2484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.172974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.2:42937)
0926 07:25:03.173738 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:03.173752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:03.173847 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:03.174107 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:03.174110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:03.174320 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:03.174428 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.174435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.175378 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.175382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.175944 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.175951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.176554 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.176558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.176663 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.177186 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:03.177198 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:03.177496 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:03.179473 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:03.179475 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:03.179477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:03.179696 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:03.179992 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:03.179994 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:03.179996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:03.180080 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:03.180619 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:03.180622 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:03.180700 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 07:25:03.180800 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:25:03.180910 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":92,"threads_started":1}
I20260926 07:25:03.181211 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.173290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42937 (local address 127.0.98.62:34689)
0926 07:25:03.173410 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:03.173414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:03.173831 (+ 417us) server_negotiation.cc:407] Connection header received
0926 07:25:03.173939 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:03.173944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:03.173998 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:03.174073 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:03.174547 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.175258 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.176059 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.176443 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.176552 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.177649 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:03.177656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:03.177662 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:03.177691 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:03.179357 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:03.179800 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:03.179803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:03.179805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:03.179897 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:03.180177 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:03.180183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:03.180184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:03.180467 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:03.180572 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:03.180891 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 07:25:03.181009 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:25:03.200326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34689
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:03.310879 2487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:03.311156 2487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:03.311273 2487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:03.315711 2487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:03.315766 2487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:03.315809 2487 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:25:03.315866 2487 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:25:03.315927 2487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:03.321630 2487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:03.322685 2487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:03.323772 2487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:03.333797 2493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:03.333894 2492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:03.333814 2487 server_base.cc:1034] running on GCE node
W20260926 07:25:03.334007 2495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:03.334453 2487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:03.335140 2487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:03.336319 2487 hybrid_clock.cc:648] HybridClock initialized: now 1790407503336303 us; error 34 us; skew 500 ppm
Sep 26 07:25:03 dist-test-slave-g04g krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:03.338896 2487 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:25:03.339708 2487 webserver.cc:460] Webserver started at http://127.0.98.3:38373/ using document root <none> and password file <none>
I20260926 07:25:03.340204 2487 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:03.340258 2487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:03.340410 2487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:03.342293 2487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "c1612b3246374887a6f1e6c837c50f82"
format_stamp: "Formatted at 2026-09-26 07:25:03 on dist-test-slave-g04g"
server_key: "a4b7112e9ddb7e022d56ea98df9b0310"
server_key_iv: "cfb62113b0faa5165b88c83f2c876f75"
server_key_version: "encryptionkey@0"
I20260926 07:25:03.342777 2487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c1612b3246374887a6f1e6c837c50f82"
format_stamp: "Formatted at 2026-09-26 07:25:03 on dist-test-slave-g04g"
server_key: "a4b7112e9ddb7e022d56ea98df9b0310"
server_key_iv: "cfb62113b0faa5165b88c83f2c876f75"
server_key_version: "encryptionkey@0"
I20260926 07:25:03.346184 2487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:25:03.348409 2502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:03.349579 2487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:03.349668 2487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "c1612b3246374887a6f1e6c837c50f82"
format_stamp: "Formatted at 2026-09-26 07:25:03 on dist-test-slave-g04g"
server_key: "a4b7112e9ddb7e022d56ea98df9b0310"
server_key_iv: "cfb62113b0faa5165b88c83f2c876f75"
server_key_version: "encryptionkey@0"
I20260926 07:25:03.349790 2487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:03.367835 2487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:03.370797 2487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:03.370986 2487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:03.372426 2487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:03.372499 2487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:03.372570 2487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:03.372620 2487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:25:03 dist-test-slave-g04g krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:03.382838 2487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43847
I20260926 07:25:03.382921 2617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43847 every 8 connection(s)
I20260926 07:25:03.383719 2487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:03.386160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 2487
I20260926 07:25:03.386272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:03.386446 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.373242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54425 (local address 127.0.98.62:34689)
0926 07:25:03.373388 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:03.373393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:03.374239 (+ 846us) server_negotiation.cc:407] Connection header received
0926 07:25:03.375430 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:03.375437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:03.375504 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:03.375610 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:03.376997 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.377884 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.379113 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.379361 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.379483 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.382769 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:03.382773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:03.382776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:03.382803 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:03.384586 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:03.385067 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:03.385076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:03.385078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:03.385154 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:03.385416 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:03.385418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:03.385419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:03.385584 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:03.385656 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:03.386141 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 07:25:03.386279 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:25:03.386567 392 external_mini_cluster.cc:1428] Setting key 8e9d3b04b7f15428077cc0b2f5b1293a
I20260926 07:25:03.387257 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.373650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:54425)
0926 07:25:03.374077 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:03.374125 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:03.375189 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:03.375799 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:03.375826 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:03.376179 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:03.376831 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.376852 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.378019 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.378023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.378955 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.378968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.379535 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.379540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.379718 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.380900 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:03.380916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:03.382502 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:03.384693 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:03.384698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:03.384710 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:03.384964 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:03.385219 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:03.385220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:03.385222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:03.385340 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:03.385674 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:03.385679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:03.385889 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 07:25:03.386059 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 07:25:03.386371 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":128,"threads_started":1}
I20260926 07:25:03.388490 2618 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34689
I20260926 07:25:03.388646 2618 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:03.389113 2618 heartbeater.cc:511] Master 127.0.98.62:34689 requested a full tablet report, sending...
I20260926 07:25:03.390061 2135 ts_manager.cc:194] Registered new tserver with Master: c1612b3246374887a6f1e6c837c50f82 (127.0.98.3:43847)
I20260926 07:25:03.390674 2135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54425
I20260926 07:25:03.397480 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.390630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.98.3:33485)
0926 07:25:03.390922 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:03.390934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:03.391011 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:03.391313 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:03.391315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:03.391518 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:03.391696 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.391704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.392456 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.392458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.393055 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.393064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.393514 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.393518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.393592 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.394175 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:03.394187 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:03.394502 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:03.396097 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:03.396099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:03.396101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:03.396303 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:03.396577 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:03.396579 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:03.396581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:03.396682 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:03.397061 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:03.397063 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:03.397138 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 07:25:03.397222 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 07:25:03.397329 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":20,"thread_start_us":107,"threads_started":1}
I20260926 07:25:03.397491 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.390720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33485 (local address 127.0.98.62:34689)
0926 07:25:03.390828 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:03.390832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:03.390981 (+ 149us) server_negotiation.cc:407] Connection header received
0926 07:25:03.391143 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:03.391148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:03.391197 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:03.391275 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:03.391827 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.392326 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.393170 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.393377 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.393461 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.394642 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:03.394645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:03.394647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:03.394667 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:03.395995 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:03.396390 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:03.396392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:03.396394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:03.396460 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:03.396799 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:03.396802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:03.396803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:03.396972 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:03.397055 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:03.397281 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 07:25:03.397382 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:25:03.400468 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:25:03 dist-test-slave-g04g krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:25:03 dist-test-slave-g04g krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:03.426574 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.418130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59588 (local address 127.0.98.62:34689)
0926 07:25:03.418268 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:03.418272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:03.418286 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:03.418327 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:03.418329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:03.418368 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:03.418440 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:03.419278 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.419792 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.420758 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.420960 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.421034 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.423502 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:03.423505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:03.423508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:03.423531 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:03.425019 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:03.425395 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:03.425397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:03.425398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:03.425469 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:03.425785 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:03.425789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:03.425791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:03.425973 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:03.426072 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:03.426272 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 07:25:03.426362 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:25:03.429579 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:25:03.432730 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:03.447234 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.442007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.0.98.3:43847)
0926 07:25:03.442448 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:03.442465 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:03.442489 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:25:03.442562 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:03.442566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:03.442755 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:03.442904 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:03.443182 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.444183 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.445661 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.446462 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.446585 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.446770 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:03.446960 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 07:25:03.447087 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":115,"threads_started":1}
I20260926 07:25:03.447234 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.442124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.0.98.1:44567)
0926 07:25:03.442610 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:03.442623 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:03.442640 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:25:03.442703 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:03.442707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:03.442860 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:03.442981 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:03.443330 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.444216 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.445885 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.446619 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.446716 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.446870 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:03.446960 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 07:25:03.447075 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":82,"threads_started":1}
I20260926 07:25:03.447288 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.441033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43847 (local address 127.0.0.1:47900)
0926 07:25:03.441716 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:03.441733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:03.441823 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:03.442897 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:03.442900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:03.442917 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:03.443002 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.443008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.444297 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.444300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.445537 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.445546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.446585 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.446588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.446755 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.446804 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 07:25:03.446920 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 07:25:03.447102 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":184,"thread_start_us":105,"threads_started":1}
I20260926 07:25:03.447288 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.441687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44567 (local address 127.0.0.1:42180)
0926 07:25:03.442463 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:03.442475 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:03.442550 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:03.443097 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:03.443100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:03.443117 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:03.443214 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.443219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.444346 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.444349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.445760 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.445767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.446733 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.446737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.446804 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.446821 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 07:25:03.446902 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:25:03.447102 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":93,"thread_start_us":108,"threads_started":1}
I20260926 07:25:03.448994 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.440988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33295 (local address 127.0.0.1:44144)
0926 07:25:03.441421 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:03.441468 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:03.441591 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:03.442393 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:03.442401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:03.442424 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:03.442565 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.442573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.444020 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.444024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.445757 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.445767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.448605 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.448608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.448697 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.448720 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:25:03.448800 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:25:03.448867 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":184,"thread_start_us":69,"threads_started":1}
I20260926 07:25:03.449041 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.441109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.0.98.2:33295)
0926 07:25:03.441511 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:03.441522 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:03.441547 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:25:03.441806 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:03.441814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:03.442029 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:03.442199 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:03.442697 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.443892 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.447445 (+ 3553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.448268 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.448397 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.448543 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:03.448843 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 07:25:03.448931 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":110,"threads_started":1}
I20260926 07:25:03.450546 2272 tablet_service.cc:1467] Processing CreateTablet for tablet e450ded2fae34a79832651a5293c8265 (DEFAULT_TABLE table=test-table [id=ddded77a85f042719174229e8637a00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:03.451617 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e450ded2fae34a79832651a5293c8265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:03.451442 2412 tablet_service.cc:1467] Processing CreateTablet for tablet e450ded2fae34a79832651a5293c8265 (DEFAULT_TABLE table=test-table [id=ddded77a85f042719174229e8637a00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:03.452607 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e450ded2fae34a79832651a5293c8265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:03.458251 2641 tablet_bootstrap.cc:492] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea: Bootstrap starting.
I20260926 07:25:03.459049 2542 tablet_service.cc:1467] Processing CreateTablet for tablet e450ded2fae34a79832651a5293c8265 (DEFAULT_TABLE table=test-table [id=ddded77a85f042719174229e8637a00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:03.460089 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e450ded2fae34a79832651a5293c8265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:03.460150 2642 tablet_bootstrap.cc:492] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589: Bootstrap starting.
I20260926 07:25:03.460853 2641 tablet_bootstrap.cc:654] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:03.461712 2641 log.cc:824] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:03.462209 2642 tablet_bootstrap.cc:654] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:03.463083 2642 log.cc:824] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:03.463989 2641 tablet_bootstrap.cc:492] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea: No bootstrap required, opened a new log
I20260926 07:25:03.464201 2641 ts_tablet_manager.cc:1397] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:25:03.464794 2642 tablet_bootstrap.cc:492] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589: No bootstrap required, opened a new log
I20260926 07:25:03.464987 2642 ts_tablet_manager.cc:1397] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:25:03.466866 2645 tablet_bootstrap.cc:492] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82: Bootstrap starting.
I20260926 07:25:03.467896 2641 raft_consensus.cc:348] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } }
I20260926 07:25:03.468137 2641 raft_consensus.cc:374] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:03.468201 2641 raft_consensus.cc:729] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e41537987f45039495bb0e26570aea, State: Initialized, Role: FOLLOWER
I20260926 07:25:03.468439 2642 raft_consensus.cc:348] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } }
I20260926 07:25:03.468678 2642 raft_consensus.cc:374] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:03.468614 2641 consensus_queue.cc:260] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } }
I20260926 07:25:03.468732 2642 raft_consensus.cc:729] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82f3787270748cda591c8ba0476f589, State: Initialized, Role: FOLLOWER
I20260926 07:25:03.468963 2645 tablet_bootstrap.cc:654] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:03.469189 2642 consensus_queue.cc:260] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } }
I20260926 07:25:03.469282 2338 heartbeater.cc:503] Master 127.0.98.62:34689 was elected leader, sending a full tablet report...
I20260926 07:25:03.469789 2645 log.cc:824] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:03.470173 2642 ts_tablet_manager.cc:1428] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 07:25:03.470508 2478 heartbeater.cc:503] Master 127.0.98.62:34689 was elected leader, sending a full tablet report...
I20260926 07:25:03.470659 2641 ts_tablet_manager.cc:1428] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:25:03.472106 2645 tablet_bootstrap.cc:492] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82: No bootstrap required, opened a new log
I20260926 07:25:03.472244 2645 ts_tablet_manager.cc:1397] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260926 07:25:03.475401 2645 raft_consensus.cc:348] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } }
I20260926 07:25:03.475589 2645 raft_consensus.cc:374] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:03.475620 2645 raft_consensus.cc:729] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1612b3246374887a6f1e6c837c50f82, State: Initialized, Role: FOLLOWER
I20260926 07:25:03.475973 2645 consensus_queue.cc:260] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } }
I20260926 07:25:03.476526 2618 heartbeater.cc:503] Master 127.0.98.62:34689 was elected leader, sending a full tablet report...
I20260926 07:25:03.476778 2645 ts_tablet_manager.cc:1428] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260926 07:25:03.507385 2647 raft_consensus.cc:482] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:03.507566 2647 raft_consensus.cc:504] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } }
I20260926 07:25:03.508606 2647 leader_election.cc:288] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1612b3246374887a6f1e6c837c50f82 (127.0.98.3:43847), b4e41537987f45039495bb0e26570aea (127.0.98.1:44567)
I20260926 07:25:03.512260 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.508982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34753 (local address 127.0.98.3:43847)
0926 07:25:03.509120 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:03.509125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:03.509141 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:25:03.509257 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:03.509274 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:03.509337 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:03.509418 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:03.509703 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.510206 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.511262 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.511707 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.511828 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.511891 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:03.512069 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 07:25:03.512133 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:25:03.512269 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.508852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43847 (local address 127.0.98.2:34753)
0926 07:25:03.508985 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:03.508998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:03.509115 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:03.509453 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:03.509457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:03.509471 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:03.509586 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.509591 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.510326 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.510333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.511135 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.511144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.511827 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.511831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.511929 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.511955 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:25:03.512057 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:25:03.512131 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 07:25:03.512863 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e450ded2fae34a79832651a5293c8265" candidate_uuid: "f82f3787270748cda591c8ba0476f589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1612b3246374887a6f1e6c837c50f82" is_pre_election: true
I20260926 07:25:03.513164 2572 raft_consensus.cc:2454] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f82f3787270748cda591c8ba0476f589 in term 0.
I20260926 07:25:03.513412 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.509281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33671 (local address 127.0.98.1:44567)
0926 07:25:03.509406 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:03.509411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:03.509609 (+ 198us) server_negotiation.cc:407] Connection header received
0926 07:25:03.509920 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:03.509922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:03.509964 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:03.510053 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:03.510312 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.510794 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.512293 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.512803 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.512930 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.512989 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:03.513220 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 07:25:03.513296 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:25:03.513448 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.509148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44567 (local address 127.0.98.2:33671)
0926 07:25:03.509526 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:03.509537 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:03.509652 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:03.510064 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:03.510067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:03.510081 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:03.510207 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.510212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.510906 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.510910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.512170 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.512178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.512942 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.512946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.513093 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.513109 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 07:25:03.513205 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 07:25:03.513307 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":8,"thread_start_us":137,"threads_started":1}
I20260926 07:25:03.513718 2364 leader_election.cc:302] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1612b3246374887a6f1e6c837c50f82, f82f3787270748cda591c8ba0476f589; no voters:
I20260926 07:25:03.513991 2647 raft_consensus.cc:2781] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:03.514106 2647 raft_consensus.cc:482] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:03.514153 2647 raft_consensus.cc:3037] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:03.514451 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e450ded2fae34a79832651a5293c8265" candidate_uuid: "f82f3787270748cda591c8ba0476f589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e41537987f45039495bb0e26570aea" is_pre_election: true
I20260926 07:25:03.514737 2290 raft_consensus.cc:2454] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f82f3787270748cda591c8ba0476f589 in term 0.
I20260926 07:25:03.515476 2647 raft_consensus.cc:504] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } }
I20260926 07:25:03.515851 2647 leader_election.cc:288] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [CANDIDATE]: Term 1 election: Requested vote from peers c1612b3246374887a6f1e6c837c50f82 (127.0.98.3:43847), b4e41537987f45039495bb0e26570aea (127.0.98.1:44567)
I20260926 07:25:03.516149 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e450ded2fae34a79832651a5293c8265" candidate_uuid: "f82f3787270748cda591c8ba0476f589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1612b3246374887a6f1e6c837c50f82"
I20260926 07:25:03.516294 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e450ded2fae34a79832651a5293c8265" candidate_uuid: "f82f3787270748cda591c8ba0476f589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e41537987f45039495bb0e26570aea"
I20260926 07:25:03.516311 2572 raft_consensus.cc:3037] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:03.516422 2290 raft_consensus.cc:3037] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:03.517557 2572 raft_consensus.cc:2454] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f82f3787270748cda591c8ba0476f589 in term 1.
I20260926 07:25:03.517627 2290 raft_consensus.cc:2454] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f82f3787270748cda591c8ba0476f589 in term 1.
I20260926 07:25:03.517948 2364 leader_election.cc:302] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1612b3246374887a6f1e6c837c50f82, f82f3787270748cda591c8ba0476f589; no voters:
I20260926 07:25:03.518124 2647 raft_consensus.cc:2781] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:03.518357 2647 raft_consensus.cc:686] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [term 1 LEADER]: Becoming Leader. State: Replica: f82f3787270748cda591c8ba0476f589, State: Running, Role: LEADER
I20260926 07:25:03.518692 2647 consensus_queue.cc:237] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } }
I20260926 07:25:03.521670 2133 catalog_manager.cc:5495] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 reported cstate change: term changed from 0 to 1, leader changed from <none> to f82f3787270748cda591c8ba0476f589 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f82f3787270748cda591c8ba0476f589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } health_report { overall_health: UNKNOWN } } }
Sep 26 07:25:03 dist-test-slave-g04g krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:03.564412 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.556492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34689 (local address 127.0.0.1:59600)
0926 07:25:03.556784 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:03.556797 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:03.556887 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:03.557102 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:03.557104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:03.557489 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:03.557585 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.557591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.558314 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.558316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.558887 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:03.558893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.559346 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.559349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:03.559418 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.560070 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:03.560083 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:03.561467 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:03.562990 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:03.562998 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:03.563003 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:03.563293 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:03.563530 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:03.563532 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:03.563533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:03.563618 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:03.563933 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:03.563938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:03.564046 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 07:25:03.564138 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:25:03.564244 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":85,"threads_started":1}
I20260926 07:25:03.564412 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:03.556582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.0.98.62:34689)
0926 07:25:03.556731 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:03.556736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:03.556868 (+ 132us) server_negotiation.cc:407] Connection header received
0926 07:25:03.556963 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:03.556966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:03.557003 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:03.557070 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:03.557721 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.558228 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.558983 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:03.559208 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:03.559280 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:03.561589 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:03.561592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:03.561594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:03.561614 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:03.562888 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:03.563380 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:03.563383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:03.563384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:03.563452 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:03.563693 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:03.563695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:03.563697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:03.563848 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:03.563918 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:03.564201 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 07:25:03.564287 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:25:03.568005 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59600:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:25:03.568737 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:03.573441 2272 tablet_service.cc:1467] Processing CreateTablet for tablet 73353dcd121d4754b6e5365c5b8e08a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=299090058d9b4460bb821823cd3a15ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:25:03.573539 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 73353dcd121d4754b6e5365c5b8e08a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=299090058d9b4460bb821823cd3a15ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:25:03.573751 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 73353dcd121d4754b6e5365c5b8e08a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=299090058d9b4460bb821823cd3a15ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:25:03.574179 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73353dcd121d4754b6e5365c5b8e08a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:03.574182 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73353dcd121d4754b6e5365c5b8e08a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:03.574383 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73353dcd121d4754b6e5365c5b8e08a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:03.576733 2645 tablet_bootstrap.cc:492] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82: Bootstrap starting.
I20260926 07:25:03.577050 2641 tablet_bootstrap.cc:492] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: Bootstrap starting.
I20260926 07:25:03.577050 2642 tablet_bootstrap.cc:492] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589: Bootstrap starting.
I20260926 07:25:03.577713 2645 tablet_bootstrap.cc:654] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:03.577970 2642 tablet_bootstrap.cc:654] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:03.577970 2641 tablet_bootstrap.cc:654] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:03.579267 2641 tablet_bootstrap.cc:492] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: No bootstrap required, opened a new log
I20260926 07:25:03.579349 2641 ts_tablet_manager.cc:1397] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:03.579396 2645 tablet_bootstrap.cc:492] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82: No bootstrap required, opened a new log
I20260926 07:25:03.579475 2645 ts_tablet_manager.cc:1397] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 07:25:03.579480 2642 tablet_bootstrap.cc:492] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589: No bootstrap required, opened a new log
I20260926 07:25:03.579545 2642 ts_tablet_manager.cc:1397] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:25:03.579805 2641 raft_consensus.cc:348] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:03.579926 2641 raft_consensus.cc:374] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:03.579973 2641 raft_consensus.cc:729] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e41537987f45039495bb0e26570aea, State: Initialized, Role: FOLLOWER
I20260926 07:25:03.579936 2642 raft_consensus.cc:348] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:03.579936 2645 raft_consensus.cc:348] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:03.580045 2642 raft_consensus.cc:374] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:03.580045 2645 raft_consensus.cc:374] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:03.580082 2642 raft_consensus.cc:729] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82f3787270748cda591c8ba0476f589, State: Initialized, Role: FOLLOWER
I20260926 07:25:03.580082 2645 raft_consensus.cc:729] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1612b3246374887a6f1e6c837c50f82, State: Initialized, Role: FOLLOWER
I20260926 07:25:03.580101 2641 consensus_queue.cc:260] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:03.580200 2642 consensus_queue.cc:260] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:03.580224 2645 consensus_queue.cc:260] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:03.580371 2641 ts_tablet_manager.cc:1428] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:03.580456 2642 ts_tablet_manager.cc:1428] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:03.580487 2645 ts_tablet_manager.cc:1428] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:25:03.580416 2646 tablet_replica.cc:439] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } }
I20260926 07:25:03.580551 2646 tablet_replica.cc:442] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:03.580658 2647 tablet_replica.cc:439] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } }
I20260926 07:25:03.580685 2649 tablet_replica.cc:439] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } }
I20260926 07:25:03.580768 2647 tablet_replica.cc:442] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:03.580801 2649 tablet_replica.cc:442] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 07:25:03.634698 2619 tablet.cc:2369] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:25:03.668648 2479 tablet.cc:2369] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:25:03.700148 2339 tablet.cc:2369] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:25:03.967922 2647 consensus_queue.cc:1035] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:03.973223 2664 consensus_queue.cc:1035] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:04.045667 2646 raft_consensus.cc:482] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:04.045840 2646 raft_consensus.cc:504] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:04.046952 2646 leader_election.cc:288] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f82f3787270748cda591c8ba0476f589 (127.0.98.2:33295), c1612b3246374887a6f1e6c837c50f82 (127.0.98.3:43847)
I20260926 07:25:04.050707 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.047236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33295 (local address 127.0.98.1:42911)
0926 07:25:04.047531 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:04.047548 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:04.047666 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:04.047946 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:04.047949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:04.047966 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:04.048086 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.048091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.048840 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.048845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.049639 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.049647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.050278 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.050281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.050375 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.050405 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:25:04.050489 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 07:25:04.050556 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
I20260926 07:25:04.050757 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.047347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42911 (local address 127.0.98.2:33295)
0926 07:25:04.047649 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:04.047654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:04.047670 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:25:04.047790 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:04.047793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:04.047833 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:04.047922 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:04.048198 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.048721 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.049760 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.050159 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.050276 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.050340 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:04.050505 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 07:25:04.050569 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":51,"threads_started":1}
I20260926 07:25:04.050848 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.047340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41969 (local address 127.0.98.3:43847)
0926 07:25:04.047669 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:04.047675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:04.047876 (+ 201us) server_negotiation.cc:407] Connection header received
0926 07:25:04.048026 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:04.048031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:04.048076 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:04.048162 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:04.048396 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.048900 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.049886 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.050299 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.050414 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.050472 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:04.050635 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 07:25:04.050705 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":87,"threads_started":1}
I20260926 07:25:04.050876 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.047236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43847 (local address 127.0.98.1:41969)
0926 07:25:04.047786 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:04.047797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:04.047916 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:04.048180 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:04.048183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:04.048198 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:04.048293 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.048298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.049027 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.049030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.049765 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.049773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.050416 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.050419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.050506 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.050534 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:25:04.050619 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 07:25:04.050685 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":219,"thread_start_us":84,"threads_started":1}
I20260926 07:25:04.051663 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73353dcd121d4754b6e5365c5b8e08a6" candidate_uuid: "b4e41537987f45039495bb0e26570aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1612b3246374887a6f1e6c837c50f82" is_pre_election: true
I20260926 07:25:04.051714 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73353dcd121d4754b6e5365c5b8e08a6" candidate_uuid: "b4e41537987f45039495bb0e26570aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82f3787270748cda591c8ba0476f589" is_pre_election: true
I20260926 07:25:04.051832 2572 raft_consensus.cc:2454] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4e41537987f45039495bb0e26570aea in term 0.
I20260926 07:25:04.051968 2430 raft_consensus.cc:2454] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4e41537987f45039495bb0e26570aea in term 0.
I20260926 07:25:04.052187 2224 leader_election.cc:302] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e41537987f45039495bb0e26570aea, c1612b3246374887a6f1e6c837c50f82; no voters:
I20260926 07:25:04.052438 2646 raft_consensus.cc:2781] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:04.052510 2646 raft_consensus.cc:482] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:04.052563 2646 raft_consensus.cc:3037] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:04.053628 2646 raft_consensus.cc:504] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:04.053956 2646 leader_election.cc:288] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [CANDIDATE]: Term 1 election: Requested vote from peers f82f3787270748cda591c8ba0476f589 (127.0.98.2:33295), c1612b3246374887a6f1e6c837c50f82 (127.0.98.3:43847)
I20260926 07:25:04.054363 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73353dcd121d4754b6e5365c5b8e08a6" candidate_uuid: "b4e41537987f45039495bb0e26570aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82f3787270748cda591c8ba0476f589"
I20260926 07:25:04.054497 2430 raft_consensus.cc:3037] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:04.054476 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73353dcd121d4754b6e5365c5b8e08a6" candidate_uuid: "b4e41537987f45039495bb0e26570aea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1612b3246374887a6f1e6c837c50f82"
I20260926 07:25:04.054580 2572 raft_consensus.cc:3037] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:04.055689 2572 raft_consensus.cc:2454] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4e41537987f45039495bb0e26570aea in term 1.
I20260926 07:25:04.055689 2430 raft_consensus.cc:2454] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4e41537987f45039495bb0e26570aea in term 1.
I20260926 07:25:04.055959 2224 leader_election.cc:302] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e41537987f45039495bb0e26570aea, c1612b3246374887a6f1e6c837c50f82; no voters:
I20260926 07:25:04.056134 2646 raft_consensus.cc:2781] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:04.056301 2646 raft_consensus.cc:686] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [term 1 LEADER]: Becoming Leader. State: Replica: b4e41537987f45039495bb0e26570aea, State: Running, Role: LEADER
I20260926 07:25:04.056608 2646 consensus_queue.cc:237] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:04.057727 2672 tablet_replica.cc:439] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: TxnStatusTablet state changed. Reason: New leader b4e41537987f45039495bb0e26570aea. Latest consensus state: current_term: 1 leader_uuid: "b4e41537987f45039495bb0e26570aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } }
I20260926 07:25:04.057840 2672 tablet_replica.cc:442] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:25:04.058099 2674 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:25:04.058338 2674 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:25:04.058912 2134 catalog_manager.cc:5495] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea reported cstate change: term changed from 0 to 1, leader changed from <none> to b4e41537987f45039495bb0e26570aea (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b4e41537987f45039495bb0e26570aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } health_report { overall_health: UNKNOWN } } }
I20260926 07:25:04.145141 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.140887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44567 (local address 127.0.0.1:42186)
0926 07:25:04.141173 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:04.141193 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:04.141321 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:04.141719 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:04.141723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:04.141732 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:25:04.141847 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.141853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.142705 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.142709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.143465 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.143476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.143994 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.143998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.144084 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.144131 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:25:04.144814 (+ 683us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:25:04.144829 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 07:25:04.144905 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 07:25:04.145004 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":100,"threads_started":1}
I20260926 07:25:04.145138 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.140980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.0.98.1:44567)
0926 07:25:04.141286 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:04.141290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:04.141306 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:25:04.141506 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:04.141510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:04.141579 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:04.141702 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:04.141963 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.142577 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.143596 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.143855 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.143973 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.144282 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:04.144707 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:04.144840 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:04.144916 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 07:25:04.144972 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":62,"threads_started":1}
I20260926 07:25:04.150310 2572 raft_consensus.cc:1261] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Refusing update from remote peer b4e41537987f45039495bb0e26570aea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:04.150468 2430 raft_consensus.cc:1261] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Refusing update from remote peer b4e41537987f45039495bb0e26570aea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:04.150835 2672 consensus_queue.cc:1035] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:04.151118 2646 consensus_queue.cc:1035] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea [LEADER]: Connected to new peer: Peer: permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:04.153764 2664 tablet_replica.cc:439] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589: TxnStatusTablet state changed. Reason: New leader b4e41537987f45039495bb0e26570aea. Latest consensus state: current_term: 1 leader_uuid: "b4e41537987f45039495bb0e26570aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } }
I20260926 07:25:04.153905 2664 tablet_replica.cc:442] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:04.154273 2649 tablet_replica.cc:439] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82: TxnStatusTablet state changed. Reason: New leader b4e41537987f45039495bb0e26570aea. Latest consensus state: current_term: 1 leader_uuid: "b4e41537987f45039495bb0e26570aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } }
I20260926 07:25:04.154381 2649 tablet_replica.cc:442] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:04.157994 2665 mvcc.cc:205] Tried to move back new op lower bound from 7333509136993509376 to 7333509136618061824. Current Snapshot: MvccSnapshot[applied={T|T < 7333509136993509376}]
I20260926 07:25:04.158077 2663 mvcc.cc:205] Tried to move back new op lower bound from 7333509136993509376 to 7333509136618061824. Current Snapshot: MvccSnapshot[applied={T|T < 7333509136993509376}]
I20260926 07:25:04.158252 2664 tablet_replica.cc:439] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4e41537987f45039495bb0e26570aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } }
I20260926 07:25:04.158350 2664 tablet_replica.cc:442] T 73353dcd121d4754b6e5365c5b8e08a6 P f82f3787270748cda591c8ba0476f589: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:04.158288 2681 tablet_replica.cc:439] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4e41537987f45039495bb0e26570aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } }
I20260926 07:25:04.158427 2681 tablet_replica.cc:442] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:25:04.158802 2681 tablet_replica.cc:439] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4e41537987f45039495bb0e26570aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } }
I20260926 07:25:04.158916 2681 tablet_replica.cc:442] T 73353dcd121d4754b6e5365c5b8e08a6 P b4e41537987f45039495bb0e26570aea: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:25:04.159232 2649 tablet_replica.cc:439] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4e41537987f45039495bb0e26570aea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } } }
I20260926 07:25:04.159359 2649 tablet_replica.cc:442] T 73353dcd121d4754b6e5365c5b8e08a6 P c1612b3246374887a6f1e6c837c50f82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:04.170421 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.166270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44158 (local address 127.0.98.2:33295)
0926 07:25:04.166383 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:04.166387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:04.166562 (+ 175us) server_negotiation.cc:407] Connection header received
0926 07:25:04.166763 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:04.166766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:04.166832 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:04.166949 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:04.167264 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.167834 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.168803 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.169066 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.169198 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.169495 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:04.169988 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:04.170132 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:04.170209 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 07:25:04.170264 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 07:25:04.181993 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.177780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37949 (local address 127.0.98.1:44567)
0926 07:25:04.178237 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:04.178242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:04.178260 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:25:04.178319 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:04.178322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:04.178371 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:04.178476 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:04.179102 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.179775 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.180766 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.181039 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.181162 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.181484 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:04.181568 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:04.181658 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:04.181788 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 07:25:04.181841 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339}
I20260926 07:25:04.182003 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.177737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44567 (local address 127.0.98.2:37949)
0926 07:25:04.178110 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:04.178130 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:04.178231 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:04.178482 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:04.178485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:04.178492 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:25:04.178621 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.178628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.179918 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.179925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.180632 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.180641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.181163 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.181166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.181269 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.181310 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:25:04.181664 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:25:04.181673 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:25:04.181763 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 07:25:04.181846 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":783,"threads_started":1}
I20260926 07:25:04.211275 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.207211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33295 (local address 127.0.98.1:46069)
0926 07:25:04.207577 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:04.207595 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:04.207707 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:04.208004 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:04.208007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:04.208016 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:25:04.208128 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.208134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.209028 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.209035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.209938 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.209962 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.210444 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.210448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.210543 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.210600 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:25:04.210925 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:25:04.210934 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:25:04.211016 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 07:25:04.211120 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":142,"threads_started":1}
I20260926 07:25:04.211248 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.207279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46069 (local address 127.0.98.2:33295)
0926 07:25:04.207420 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:04.207426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:04.207676 (+ 250us) server_negotiation.cc:407] Connection header received
0926 07:25:04.207827 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:04.207831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:04.207889 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:04.207987 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:04.208295 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.208901 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.210080 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.210317 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.210434 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.210750 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:04.210830 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:04.210920 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:04.211048 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 07:25:04.211098 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:25:04.238531 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:04.238718 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:04.241081 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3: Sending DeleteTablet for 3 replicas of tablet e450ded2fae34a79832651a5293c8265
I20260926 07:25:04.241729 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet e450ded2fae34a79832651a5293c8265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44144
I20260926 07:25:04.241855 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet e450ded2fae34a79832651a5293c8265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42180
I20260926 07:25:04.241984 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet e450ded2fae34a79832651a5293c8265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47900
I20260926 07:25:04.242244 2695 tablet_replica.cc:330] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589: stopping tablet replica
I20260926 07:25:04.242439 2696 tablet_replica.cc:330] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea: stopping tablet replica
I20260926 07:25:04.242573 2695 raft_consensus.cc:2229] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:25:04.242868 2696 raft_consensus.cc:2229] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:04.242820 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:25:04.242936 2695 raft_consensus.cc:2258] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:04.243255 2696 raft_consensus.cc:2258] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 07:25:04.243257 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:04.244040 2697 tablet_replica.cc:330] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82: stopping tablet replica
I20260926 07:25:04.244216 2695 ts_tablet_manager.cc:1905] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:04.244292 2697 raft_consensus.cc:2229] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:04.244489 2697 raft_consensus.cc:2258] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:04.245635 2697 ts_tablet_manager.cc:1905] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:04.247572 2695 ts_tablet_manager.cc:1918] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:04.247674 2695 log.cc:1196] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/e450ded2fae34a79832651a5293c8265
I20260926 07:25:04.248057 2695 ts_tablet_manager.cc:1939] T e450ded2fae34a79832651a5293c8265 P f82f3787270748cda591c8ba0476f589: Deleting consensus metadata
I20260926 07:25:04.248473 2697 ts_tablet_manager.cc:1918] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:04.248562 2697 log.cc:1196] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/e450ded2fae34a79832651a5293c8265
I20260926 07:25:04.248858 2697 ts_tablet_manager.cc:1939] T e450ded2fae34a79832651a5293c8265 P c1612b3246374887a6f1e6c837c50f82: Deleting consensus metadata
I20260926 07:25:04.249303 2122 catalog_manager.cc:4841] TS f82f3787270748cda591c8ba0476f589 (127.0.98.2:33295): tablet e450ded2fae34a79832651a5293c8265 (table test-table [id=ddded77a85f042719174229e8637a00c]) successfully deleted
I20260926 07:25:04.250053 2120 catalog_manager.cc:4841] TS c1612b3246374887a6f1e6c837c50f82 (127.0.98.3:43847): tablet e450ded2fae34a79832651a5293c8265 (table test-table [id=ddded77a85f042719174229e8637a00c]) successfully deleted
I20260926 07:25:04.250257 2272 tablet_service.cc:1467] Processing CreateTablet for tablet b6c0732aa4d94f16857191776d2953fe (DEFAULT_TABLE table=test-table [id=402d0529e35946879b03fe30e441ca14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:04.250339 2412 tablet_service.cc:1467] Processing CreateTablet for tablet b6c0732aa4d94f16857191776d2953fe (DEFAULT_TABLE table=test-table [id=402d0529e35946879b03fe30e441ca14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:04.250521 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c0732aa4d94f16857191776d2953fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:04.250523 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c0732aa4d94f16857191776d2953fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:04.252770 2698 tablet_bootstrap.cc:492] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589: Bootstrap starting.
I20260926 07:25:04.253105 2542 tablet_service.cc:1467] Processing CreateTablet for tablet b6c0732aa4d94f16857191776d2953fe (DEFAULT_TABLE table=test-table [id=402d0529e35946879b03fe30e441ca14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:04.253343 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6c0732aa4d94f16857191776d2953fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:04.253890 2698 tablet_bootstrap.cc:654] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:04.254783 2696 ts_tablet_manager.cc:1905] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:04.255652 2701 tablet_bootstrap.cc:492] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82: Bootstrap starting.
I20260926 07:25:04.256635 2701 tablet_bootstrap.cc:654] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:04.257680 2698 tablet_bootstrap.cc:492] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589: No bootstrap required, opened a new log
I20260926 07:25:04.257774 2698 ts_tablet_manager.cc:1397] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 07:25:04.258132 2696 ts_tablet_manager.cc:1918] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:04.258231 2696 log.cc:1196] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/wal/wals/e450ded2fae34a79832651a5293c8265
I20260926 07:25:04.258276 2698 raft_consensus.cc:348] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:04.258400 2698 raft_consensus.cc:374] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:04.258443 2698 raft_consensus.cc:729] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82f3787270748cda591c8ba0476f589, State: Initialized, Role: FOLLOWER
I20260926 07:25:04.258576 2696 ts_tablet_manager.cc:1939] T e450ded2fae34a79832651a5293c8265 P b4e41537987f45039495bb0e26570aea: Deleting consensus metadata
I20260926 07:25:04.258579 2698 consensus_queue.cc:260] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:04.258910 2698 ts_tablet_manager.cc:1428] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:04.259560 2699 tablet_bootstrap.cc:492] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea: Bootstrap starting.
I20260926 07:25:04.260245 2699 tablet_bootstrap.cc:654] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:04.260572 2701 tablet_bootstrap.cc:492] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82: No bootstrap required, opened a new log
I20260926 07:25:04.260664 2701 ts_tablet_manager.cc:1397] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 07:25:04.261122 2701 raft_consensus.cc:348] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:04.261238 2701 raft_consensus.cc:374] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:04.261278 2701 raft_consensus.cc:729] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1612b3246374887a6f1e6c837c50f82, State: Initialized, Role: FOLLOWER
I20260926 07:25:04.261421 2120 catalog_manager.cc:4841] TS b4e41537987f45039495bb0e26570aea (127.0.98.1:44567): tablet e450ded2fae34a79832651a5293c8265 (table test-table [id=ddded77a85f042719174229e8637a00c]) successfully deleted
I20260926 07:25:04.261436 2701 consensus_queue.cc:260] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:04.261693 2701 ts_tablet_manager.cc:1428] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:04.262828 2699 tablet_bootstrap.cc:492] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea: No bootstrap required, opened a new log
I20260926 07:25:04.262924 2699 ts_tablet_manager.cc:1397] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 07:25:04.263372 2699 raft_consensus.cc:348] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:04.263496 2699 raft_consensus.cc:374] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:04.263533 2699 raft_consensus.cc:729] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e41537987f45039495bb0e26570aea, State: Initialized, Role: FOLLOWER
I20260926 07:25:04.263650 2699 consensus_queue.cc:260] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:04.264014 2699 ts_tablet_manager.cc:1428] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:04.474311 2649 raft_consensus.cc:482] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:04.474486 2649 raft_consensus.cc:504] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:04.475711 2649 leader_election.cc:288] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4e41537987f45039495bb0e26570aea (127.0.98.1:44567), f82f3787270748cda591c8ba0476f589 (127.0.98.2:33295)
I20260926 07:25:04.479664 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.476060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43515 (local address 127.0.98.2:33295)
0926 07:25:04.476206 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:04.476211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:04.476434 (+ 223us) server_negotiation.cc:407] Connection header received
0926 07:25:04.476593 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:04.476598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:04.476643 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:04.476727 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:04.476976 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.477462 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.478596 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.479094 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.479215 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.479275 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:04.479458 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 07:25:04.479540 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:25:04.479717 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.475920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44567 (local address 127.0.98.3:48279)
0926 07:25:04.476273 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:04.476288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:04.476434 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:04.476706 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:04.476709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:04.476723 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:04.476854 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.476859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.477574 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.477577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.478484 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.478492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.479370 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.479373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.479460 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.479479 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 07:25:04.479543 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 07:25:04.479595 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":24,"spinlock_wait_cycles":21760,"thread_start_us":117,"threads_started":1}
I20260926 07:25:04.479776 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.476039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48279 (local address 127.0.98.1:44567)
0926 07:25:04.476251 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:04.476256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:04.476379 (+ 123us) server_negotiation.cc:407] Connection header received
0926 07:25:04.476547 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:04.476550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:04.476613 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:04.476697 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:04.476959 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.477466 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.478665 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.479194 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.479257 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.479307 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:04.479582 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 07:25:04.479658 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260926 07:25:04.480229 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.475987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33295 (local address 127.0.98.3:43515)
0926 07:25:04.476339 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:04.476350 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:04.476486 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:04.476735 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:04.476738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:04.476775 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:04.476871 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.476876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.477635 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.477638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.478472 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.478480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.479226 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.479229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.479329 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.479355 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:25:04.479430 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:25:04.479499 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":57,"spinlock_wait_cycles":768,"thread_start_us":123,"threads_started":1}
I20260926 07:25:04.480396 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c0732aa4d94f16857191776d2953fe" candidate_uuid: "c1612b3246374887a6f1e6c837c50f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e41537987f45039495bb0e26570aea" is_pre_election: true
I20260926 07:25:04.480549 2290 raft_consensus.cc:2454] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1612b3246374887a6f1e6c837c50f82 in term 0.
I20260926 07:25:04.480692 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c0732aa4d94f16857191776d2953fe" candidate_uuid: "c1612b3246374887a6f1e6c837c50f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82f3787270748cda591c8ba0476f589" is_pre_election: true
I20260926 07:25:04.480829 2430 raft_consensus.cc:2454] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1612b3246374887a6f1e6c837c50f82 in term 0.
I20260926 07:25:04.480926 2504 leader_election.cc:302] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e41537987f45039495bb0e26570aea, c1612b3246374887a6f1e6c837c50f82; no voters:
I20260926 07:25:04.481194 2649 raft_consensus.cc:2781] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:04.481269 2649 raft_consensus.cc:482] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:04.481324 2649 raft_consensus.cc:3037] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:04.482244 2649 raft_consensus.cc:504] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:04.482617 2649 leader_election.cc:288] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [CANDIDATE]: Term 1 election: Requested vote from peers b4e41537987f45039495bb0e26570aea (127.0.98.1:44567), f82f3787270748cda591c8ba0476f589 (127.0.98.2:33295)
I20260926 07:25:04.482939 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c0732aa4d94f16857191776d2953fe" candidate_uuid: "c1612b3246374887a6f1e6c837c50f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e41537987f45039495bb0e26570aea"
I20260926 07:25:04.482991 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6c0732aa4d94f16857191776d2953fe" candidate_uuid: "c1612b3246374887a6f1e6c837c50f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82f3787270748cda591c8ba0476f589"
I20260926 07:25:04.483110 2290 raft_consensus.cc:3037] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:04.483196 2430 raft_consensus.cc:3037] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:04.484236 2290 raft_consensus.cc:2454] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1612b3246374887a6f1e6c837c50f82 in term 1.
I20260926 07:25:04.484381 2430 raft_consensus.cc:2454] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1612b3246374887a6f1e6c837c50f82 in term 1.
I20260926 07:25:04.484508 2504 leader_election.cc:302] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e41537987f45039495bb0e26570aea, c1612b3246374887a6f1e6c837c50f82; no voters:
I20260926 07:25:04.484726 2649 raft_consensus.cc:2781] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:04.484902 2649 raft_consensus.cc:686] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [term 1 LEADER]: Becoming Leader. State: Replica: c1612b3246374887a6f1e6c837c50f82, State: Running, Role: LEADER
I20260926 07:25:04.485147 2649 consensus_queue.cc:237] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:04.487202 2134 catalog_manager.cc:5495] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1612b3246374887a6f1e6c837c50f82 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c1612b3246374887a6f1e6c837c50f82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } health_report { overall_health: HEALTHY } } }
I20260926 07:25:04.554715 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.550635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.0.98.3:43847)
0926 07:25:04.550752 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:04.550765 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:04.550780 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:25:04.550984 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:04.550987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:04.551084 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:04.551203 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:04.551497 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.552081 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.553118 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.553407 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.553521 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.553781 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:04.554304 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:04.554445 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:04.554524 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 07:25:04.554576 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:25:04.564855 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.561428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44567 (local address 127.0.98.3:55689)
0926 07:25:04.561764 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:04.561777 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:04.561891 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:04.562151 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:04.562153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:04.562161 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:25:04.562267 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.562275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.562984 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.562988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.563657 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.563665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.564111 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.564114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.564206 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.564258 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:25:04.564572 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:25:04.564578 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:25:04.564638 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 07:25:04.564711 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":79,"threads_started":1}
I20260926 07:25:04.564841 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.561516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55689 (local address 127.0.98.1:44567)
0926 07:25:04.561665 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:04.561670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:04.561846 (+ 176us) server_negotiation.cc:407] Connection header received
0926 07:25:04.562001 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:04.562005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:04.562059 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:04.562142 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:04.562379 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.562873 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.563789 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.564002 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.564110 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.564404 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:04.564476 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:04.564557 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:04.564647 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 07:25:04.564696 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:25:04.570534 2430 raft_consensus.cc:1261] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Refusing update from remote peer c1612b3246374887a6f1e6c837c50f82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:04.570552 2290 raft_consensus.cc:1261] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Refusing update from remote peer c1612b3246374887a6f1e6c837c50f82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:04.570986 2649 consensus_queue.cc:1035] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:04.571242 2706 consensus_queue.cc:1035] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:04.588074 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.584757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35541 (local address 127.0.98.3:43847)
0926 07:25:04.584853 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:04.584858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:04.584969 (+ 111us) server_negotiation.cc:407] Connection header received
0926 07:25:04.585092 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:04.585095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:04.585148 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:04.585226 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:04.585463 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.586067 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.586844 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.587125 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.587237 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.587498 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:04.587586 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:04.587661 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:04.587840 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 07:25:04.587888 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 07:25:04.588081 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:04.584708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43847 (local address 127.0.98.1:35541)
0926 07:25:04.584884 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:04.584898 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:04.584992 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:04.585247 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:04.585250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:04.585258 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:25:04.585356 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.585365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.586167 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.586170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.586746 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:04.586754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:04.587249 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:04.587252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:04.587348 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:04.587382 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:25:04.587714 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:25:04.587722 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:25:04.587812 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 07:25:04.587899 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
W20260926 07:25:04.588959 2665 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 07:25:06.156741 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:06.156900 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:06.159109 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3: Sending DeleteTablet for 3 replicas of tablet b6c0732aa4d94f16857191776d2953fe
I20260926 07:25:06.159751 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c0732aa4d94f16857191776d2953fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42180
I20260926 07:25:06.159751 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c0732aa4d94f16857191776d2953fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47900
I20260926 07:25:06.159861 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c0732aa4d94f16857191776d2953fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44144
I20260926 07:25:06.160126 2738 tablet_replica.cc:330] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea: stopping tablet replica
I20260926 07:25:06.160180 2737 tablet_replica.cc:330] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82: stopping tablet replica
I20260926 07:25:06.160247 2739 tablet_replica.cc:330] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589: stopping tablet replica
I20260926 07:25:06.160344 2738 raft_consensus.cc:2229] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:06.160382 2737 raft_consensus.cc:2229] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:25:06.160380 2739 raft_consensus.cc:2229] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:06.160513 2738 raft_consensus.cc:2258] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:06.160552 2739 raft_consensus.cc:2258] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:06.160739 2737 raft_consensus.cc:2258] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:06.161221 2738 ts_tablet_manager.cc:1905] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:06.161366 2737 ts_tablet_manager.cc:1905] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:06.162854 2739 ts_tablet_manager.cc:1905] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:06.164387 2737 ts_tablet_manager.cc:1918] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:25:06.164513 2737 log.cc:1196] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/b6c0732aa4d94f16857191776d2953fe
I20260926 07:25:06.164885 2737 ts_tablet_manager.cc:1939] T b6c0732aa4d94f16857191776d2953fe P c1612b3246374887a6f1e6c837c50f82: Deleting consensus metadata
I20260926 07:25:06.165066 2738 ts_tablet_manager.cc:1918] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:25:06.165189 2738 log.cc:1196] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/wal/wals/b6c0732aa4d94f16857191776d2953fe
I20260926 07:25:06.165529 2738 ts_tablet_manager.cc:1939] T b6c0732aa4d94f16857191776d2953fe P b4e41537987f45039495bb0e26570aea: Deleting consensus metadata
I20260926 07:25:06.165648 2739 ts_tablet_manager.cc:1918] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:25:06.165752 2739 log.cc:1196] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/b6c0732aa4d94f16857191776d2953fe
I20260926 07:25:06.165756 2120 catalog_manager.cc:4841] TS c1612b3246374887a6f1e6c837c50f82 (127.0.98.3:43847): tablet b6c0732aa4d94f16857191776d2953fe (table test-table [id=402d0529e35946879b03fe30e441ca14]) successfully deleted
I20260926 07:25:06.166028 2739 ts_tablet_manager.cc:1939] T b6c0732aa4d94f16857191776d2953fe P f82f3787270748cda591c8ba0476f589: Deleting consensus metadata
I20260926 07:25:06.166178 2120 catalog_manager.cc:4841] TS b4e41537987f45039495bb0e26570aea (127.0.98.1:44567): tablet b6c0732aa4d94f16857191776d2953fe (table test-table [id=402d0529e35946879b03fe30e441ca14]) successfully deleted
I20260926 07:25:06.166692 2122 catalog_manager.cc:4841] TS f82f3787270748cda591c8ba0476f589 (127.0.98.2:33295): tablet b6c0732aa4d94f16857191776d2953fe (table test-table [id=402d0529e35946879b03fe30e441ca14]) successfully deleted
Sep 26 07:25:06 dist-test-slave-g04g krb5kdc[2090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407506, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:25:06 dist-test-slave-g04g krb5kdc[2090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407506, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:06.195268 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:06.182387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59608 (local address 127.0.98.62:34689)
0926 07:25:06.182704 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:06.182711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:06.182726 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:25:06.182819 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:06.182822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:06.182865 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:06.182948 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:06.183854 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:06.184503 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:06.185356 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:06.185622 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:06.185719 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:06.191968 (+ 6249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:06.191971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:06.191973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:06.191999 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:06.193539 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:06.193963 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:06.193967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:06.193969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:06.194050 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:06.194328 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:06.194333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:06.194335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:06.194527 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:06.194663 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:06.194902 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 07:25:06.195019 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":134,"threads_started":1}
I20260926 07:25:06.197921 2135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:25:06.198319 2135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:06.202986 2272 tablet_service.cc:1467] Processing CreateTablet for tablet f2c67f5a56d845ac8919ab4dd9a3ac26 (DEFAULT_TABLE table=test-table [id=fdf5d22de6714dbdb30e805e99f6ae21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:06.203174 2412 tablet_service.cc:1467] Processing CreateTablet for tablet f2c67f5a56d845ac8919ab4dd9a3ac26 (DEFAULT_TABLE table=test-table [id=fdf5d22de6714dbdb30e805e99f6ae21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:06.203332 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c67f5a56d845ac8919ab4dd9a3ac26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:06.203425 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c67f5a56d845ac8919ab4dd9a3ac26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:06.205840 2748 tablet_bootstrap.cc:492] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589: Bootstrap starting.
I20260926 07:25:06.206030 2542 tablet_service.cc:1467] Processing CreateTablet for tablet f2c67f5a56d845ac8919ab4dd9a3ac26 (DEFAULT_TABLE table=test-table [id=fdf5d22de6714dbdb30e805e99f6ae21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:06.206333 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2c67f5a56d845ac8919ab4dd9a3ac26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:06.206790 2748 tablet_bootstrap.cc:654] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:06.207492 2749 tablet_bootstrap.cc:492] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea: Bootstrap starting.
I20260926 07:25:06.208077 2748 tablet_bootstrap.cc:492] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589: No bootstrap required, opened a new log
I20260926 07:25:06.208173 2748 ts_tablet_manager.cc:1397] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:06.208400 2749 tablet_bootstrap.cc:654] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:06.208719 2748 raft_consensus.cc:348] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.208945 2748 raft_consensus.cc:374] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:06.208992 2751 tablet_bootstrap.cc:492] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82: Bootstrap starting.
I20260926 07:25:06.209012 2748 raft_consensus.cc:729] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82f3787270748cda591c8ba0476f589, State: Initialized, Role: FOLLOWER
I20260926 07:25:06.209208 2748 consensus_queue.cc:260] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.209659 2748 ts_tablet_manager.cc:1428] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:06.209794 2749 tablet_bootstrap.cc:492] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea: No bootstrap required, opened a new log
I20260926 07:25:06.209856 2751 tablet_bootstrap.cc:654] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:06.209892 2749 ts_tablet_manager.cc:1397] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:06.210307 2749 raft_consensus.cc:348] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.210464 2749 raft_consensus.cc:374] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:06.210500 2749 raft_consensus.cc:729] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e41537987f45039495bb0e26570aea, State: Initialized, Role: FOLLOWER
I20260926 07:25:06.210641 2749 consensus_queue.cc:260] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.210917 2749 ts_tablet_manager.cc:1428] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:06.211012 2751 tablet_bootstrap.cc:492] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82: No bootstrap required, opened a new log
I20260926 07:25:06.211143 2751 ts_tablet_manager.cc:1397] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:06.211535 2751 raft_consensus.cc:348] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.211651 2751 raft_consensus.cc:374] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:06.211720 2751 raft_consensus.cc:729] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1612b3246374887a6f1e6c837c50f82, State: Initialized, Role: FOLLOWER
I20260926 07:25:06.211860 2751 consensus_queue.cc:260] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.212605 2751 ts_tablet_manager.cc:1428] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:25:06.281368 2714 raft_consensus.cc:482] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:06.281505 2714 raft_consensus.cc:504] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.281950 2714 leader_election.cc:288] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4e41537987f45039495bb0e26570aea (127.0.98.1:44567), f82f3787270748cda591c8ba0476f589 (127.0.98.2:33295)
I20260926 07:25:06.282351 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c67f5a56d845ac8919ab4dd9a3ac26" candidate_uuid: "c1612b3246374887a6f1e6c837c50f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e41537987f45039495bb0e26570aea" is_pre_election: true
I20260926 07:25:06.282351 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c67f5a56d845ac8919ab4dd9a3ac26" candidate_uuid: "c1612b3246374887a6f1e6c837c50f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82f3787270748cda591c8ba0476f589" is_pre_election: true
I20260926 07:25:06.282516 2432 raft_consensus.cc:2454] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1612b3246374887a6f1e6c837c50f82 in term 0.
I20260926 07:25:06.282516 2290 raft_consensus.cc:2454] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1612b3246374887a6f1e6c837c50f82 in term 0.
I20260926 07:25:06.282832 2504 leader_election.cc:302] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b4e41537987f45039495bb0e26570aea, c1612b3246374887a6f1e6c837c50f82, f82f3787270748cda591c8ba0476f589; no voters:
I20260926 07:25:06.283108 2714 raft_consensus.cc:2781] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:06.283198 2714 raft_consensus.cc:482] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:06.283267 2714 raft_consensus.cc:3037] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:06.284287 2714 raft_consensus.cc:504] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.284566 2714 leader_election.cc:288] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [CANDIDATE]: Term 1 election: Requested vote from peers b4e41537987f45039495bb0e26570aea (127.0.98.1:44567), f82f3787270748cda591c8ba0476f589 (127.0.98.2:33295)
I20260926 07:25:06.284910 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c67f5a56d845ac8919ab4dd9a3ac26" candidate_uuid: "c1612b3246374887a6f1e6c837c50f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82f3787270748cda591c8ba0476f589"
I20260926 07:25:06.284910 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2c67f5a56d845ac8919ab4dd9a3ac26" candidate_uuid: "c1612b3246374887a6f1e6c837c50f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e41537987f45039495bb0e26570aea"
I20260926 07:25:06.285037 2290 raft_consensus.cc:3037] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:06.285037 2432 raft_consensus.cc:3037] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:06.286134 2290 raft_consensus.cc:2454] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1612b3246374887a6f1e6c837c50f82 in term 1.
I20260926 07:25:06.286134 2432 raft_consensus.cc:2454] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1612b3246374887a6f1e6c837c50f82 in term 1.
I20260926 07:25:06.286486 2506 leader_election.cc:302] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1612b3246374887a6f1e6c837c50f82, f82f3787270748cda591c8ba0476f589; no voters:
I20260926 07:25:06.286667 2714 raft_consensus.cc:2781] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:06.286774 2714 raft_consensus.cc:686] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [term 1 LEADER]: Becoming Leader. State: Replica: c1612b3246374887a6f1e6c837c50f82, State: Running, Role: LEADER
I20260926 07:25:06.286916 2714 consensus_queue.cc:237] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.288800 2133 catalog_manager.cc:5495] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1612b3246374887a6f1e6c837c50f82 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c1612b3246374887a6f1e6c837c50f82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } health_report { overall_health: HEALTHY } } }
I20260926 07:25:06.312819 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:06.309590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.0.98.3:43847)
0926 07:25:06.309871 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:06.309881 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:06.309894 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:25:06.309941 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:06.309943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:06.309981 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:06.310073 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:06.310327 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:06.310887 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:06.311766 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:06.311999 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:06.312114 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:06.312363 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:06.312447 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:06.312526 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:06.312612 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 07:25:06.312664 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":108,"threads_started":1}
I20260926 07:25:06.318583 2432 raft_consensus.cc:1261] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Refusing update from remote peer c1612b3246374887a6f1e6c837c50f82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:06.318653 2290 raft_consensus.cc:1261] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Refusing update from remote peer c1612b3246374887a6f1e6c837c50f82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:06.319060 2728 consensus_queue.cc:1035] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:06.319265 2714 consensus_queue.cc:1035] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:06.324647 2716 mvcc.cc:205] Tried to move back new op lower bound from 7333509145878024192 to 7333509145752772608. Current Snapshot: MvccSnapshot[applied={T|T < 7333509145878024192}]
I20260926 07:25:06.349475 2133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:06.349612 2133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:06.351625 2133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3: Sending DeleteTablet for 3 replicas of tablet f2c67f5a56d845ac8919ab4dd9a3ac26
I20260926 07:25:06.352097 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c67f5a56d845ac8919ab4dd9a3ac26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42180
I20260926 07:25:06.352273 2738 tablet_replica.cc:330] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea: stopping tablet replica
I20260926 07:25:06.352283 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c67f5a56d845ac8919ab4dd9a3ac26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44144
I20260926 07:25:06.352406 2738 raft_consensus.cc:2229] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:06.352512 2739 tablet_replica.cc:330] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589: stopping tablet replica
I20260926 07:25:06.352586 2738 raft_consensus.cc:2258] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:06.352643 2739 raft_consensus.cc:2229] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:06.352797 2739 raft_consensus.cc:2258] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:06.353009 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet f2c67f5a56d845ac8919ab4dd9a3ac26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47900
I20260926 07:25:06.353052 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:25:06.353221 2737 tablet_replica.cc:330] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82: stopping tablet replica
I20260926 07:25:06.353358 2738 ts_tablet_manager.cc:1905] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:06.353389 2737 raft_consensus.cc:2229] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [term 1 LEADER]: Raft consensus shutting down.
W20260926 07:25:06.353416 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:06.353586 2737 raft_consensus.cc:2258] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:06.354424 2737 ts_tablet_manager.cc:1905] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:06.354797 2739 ts_tablet_manager.cc:1905] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:06.355938 2738 ts_tablet_manager.cc:1918] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:06.356022 2738 log.cc:1196] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-0/wal/wals/f2c67f5a56d845ac8919ab4dd9a3ac26
I20260926 07:25:06.356276 2738 ts_tablet_manager.cc:1939] T f2c67f5a56d845ac8919ab4dd9a3ac26 P b4e41537987f45039495bb0e26570aea: Deleting consensus metadata
I20260926 07:25:06.356951 2120 catalog_manager.cc:4841] TS b4e41537987f45039495bb0e26570aea (127.0.98.1:44567): tablet f2c67f5a56d845ac8919ab4dd9a3ac26 (table test-table [id=fdf5d22de6714dbdb30e805e99f6ae21]) successfully deleted
I20260926 07:25:06.357309 2739 ts_tablet_manager.cc:1918] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:06.357358 2739 log.cc:1196] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/f2c67f5a56d845ac8919ab4dd9a3ac26
I20260926 07:25:06.357438 2737 ts_tablet_manager.cc:1918] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:06.357522 2737 log.cc:1196] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/f2c67f5a56d845ac8919ab4dd9a3ac26
I20260926 07:25:06.357584 2739 ts_tablet_manager.cc:1939] T f2c67f5a56d845ac8919ab4dd9a3ac26 P f82f3787270748cda591c8ba0476f589: Deleting consensus metadata
I20260926 07:25:06.357903 2737 ts_tablet_manager.cc:1939] T f2c67f5a56d845ac8919ab4dd9a3ac26 P c1612b3246374887a6f1e6c837c50f82: Deleting consensus metadata
I20260926 07:25:06.358767 2272 tablet_service.cc:1467] Processing CreateTablet for tablet 776b246a8b4242eda1e6629b8e0414bc (DEFAULT_TABLE table=test-table [id=8410d9cd26ac4a6d910415d3dff1a695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:06.359071 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776b246a8b4242eda1e6629b8e0414bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:06.359146 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 776b246a8b4242eda1e6629b8e0414bc (DEFAULT_TABLE table=test-table [id=8410d9cd26ac4a6d910415d3dff1a695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:06.359423 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776b246a8b4242eda1e6629b8e0414bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:06.359422 2122 catalog_manager.cc:4841] TS f82f3787270748cda591c8ba0476f589 (127.0.98.2:33295): tablet f2c67f5a56d845ac8919ab4dd9a3ac26 (table test-table [id=fdf5d22de6714dbdb30e805e99f6ae21]) successfully deleted
I20260926 07:25:06.361104 2749 tablet_bootstrap.cc:492] T 776b246a8b4242eda1e6629b8e0414bc P b4e41537987f45039495bb0e26570aea: Bootstrap starting.
I20260926 07:25:06.361328 2120 catalog_manager.cc:4841] TS c1612b3246374887a6f1e6c837c50f82 (127.0.98.3:43847): tablet f2c67f5a56d845ac8919ab4dd9a3ac26 (table test-table [id=fdf5d22de6714dbdb30e805e99f6ae21]) successfully deleted
I20260926 07:25:06.361508 2748 tablet_bootstrap.cc:492] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589: Bootstrap starting.
I20260926 07:25:06.362102 2749 tablet_bootstrap.cc:654] T 776b246a8b4242eda1e6629b8e0414bc P b4e41537987f45039495bb0e26570aea: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:06.362469 2748 tablet_bootstrap.cc:654] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:06.363207 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 776b246a8b4242eda1e6629b8e0414bc (DEFAULT_TABLE table=test-table [id=8410d9cd26ac4a6d910415d3dff1a695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:06.363412 2749 tablet_bootstrap.cc:492] T 776b246a8b4242eda1e6629b8e0414bc P b4e41537987f45039495bb0e26570aea: No bootstrap required, opened a new log
I20260926 07:25:06.363492 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776b246a8b4242eda1e6629b8e0414bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:06.363502 2749 ts_tablet_manager.cc:1397] T 776b246a8b4242eda1e6629b8e0414bc P b4e41537987f45039495bb0e26570aea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:06.363992 2749 raft_consensus.cc:348] T 776b246a8b4242eda1e6629b8e0414bc P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.364112 2749 raft_consensus.cc:374] T 776b246a8b4242eda1e6629b8e0414bc P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:06.364149 2749 raft_consensus.cc:729] T 776b246a8b4242eda1e6629b8e0414bc P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e41537987f45039495bb0e26570aea, State: Initialized, Role: FOLLOWER
I20260926 07:25:06.364295 2749 consensus_queue.cc:260] T 776b246a8b4242eda1e6629b8e0414bc P b4e41537987f45039495bb0e26570aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.364622 2749 ts_tablet_manager.cc:1428] T 776b246a8b4242eda1e6629b8e0414bc P b4e41537987f45039495bb0e26570aea: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:25:06.365518 2748 tablet_bootstrap.cc:492] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589: No bootstrap required, opened a new log
I20260926 07:25:06.365602 2748 ts_tablet_manager.cc:1397] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 07:25:06.366011 2748 raft_consensus.cc:348] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.366128 2748 raft_consensus.cc:374] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:06.366170 2748 raft_consensus.cc:729] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82f3787270748cda591c8ba0476f589, State: Initialized, Role: FOLLOWER
I20260926 07:25:06.366237 2751 tablet_bootstrap.cc:492] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82: Bootstrap starting.
I20260926 07:25:06.366290 2748 consensus_queue.cc:260] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.366566 2748 ts_tablet_manager.cc:1428] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:06.367295 2751 tablet_bootstrap.cc:654] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:06.368860 2751 tablet_bootstrap.cc:492] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82: No bootstrap required, opened a new log
I20260926 07:25:06.368945 2751 ts_tablet_manager.cc:1397] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 07:25:06.369520 2751 raft_consensus.cc:348] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.369618 2751 raft_consensus.cc:374] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:06.369644 2751 raft_consensus.cc:729] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1612b3246374887a6f1e6c837c50f82, State: Initialized, Role: FOLLOWER
I20260926 07:25:06.369789 2751 consensus_queue.cc:260] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.370102 2751 ts_tablet_manager.cc:1428] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:25:06.408141 2714 raft_consensus.cc:482] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:06.408277 2714 raft_consensus.cc:504] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.408660 2714 leader_election.cc:288] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4e41537987f45039495bb0e26570aea (127.0.98.1:44567), f82f3787270748cda591c8ba0476f589 (127.0.98.2:33295)
I20260926 07:25:06.409195 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776b246a8b4242eda1e6629b8e0414bc" candidate_uuid: "c1612b3246374887a6f1e6c837c50f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82f3787270748cda591c8ba0476f589" is_pre_election: true
I20260926 07:25:06.409251 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776b246a8b4242eda1e6629b8e0414bc" candidate_uuid: "c1612b3246374887a6f1e6c837c50f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e41537987f45039495bb0e26570aea" is_pre_election: true
I20260926 07:25:06.409358 2432 raft_consensus.cc:2454] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1612b3246374887a6f1e6c837c50f82 in term 0.
I20260926 07:25:06.409502 2290 raft_consensus.cc:2454] T 776b246a8b4242eda1e6629b8e0414bc P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1612b3246374887a6f1e6c837c50f82 in term 0.
I20260926 07:25:06.409662 2506 leader_election.cc:302] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1612b3246374887a6f1e6c837c50f82, f82f3787270748cda591c8ba0476f589; no voters:
I20260926 07:25:06.409891 2714 raft_consensus.cc:2781] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:06.409973 2714 raft_consensus.cc:482] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:06.410025 2714 raft_consensus.cc:3037] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:06.411085 2714 raft_consensus.cc:504] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.411427 2714 leader_election.cc:288] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [CANDIDATE]: Term 1 election: Requested vote from peers b4e41537987f45039495bb0e26570aea (127.0.98.1:44567), f82f3787270748cda591c8ba0476f589 (127.0.98.2:33295)
I20260926 07:25:06.411727 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776b246a8b4242eda1e6629b8e0414bc" candidate_uuid: "c1612b3246374887a6f1e6c837c50f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e41537987f45039495bb0e26570aea"
I20260926 07:25:06.411787 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776b246a8b4242eda1e6629b8e0414bc" candidate_uuid: "c1612b3246374887a6f1e6c837c50f82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82f3787270748cda591c8ba0476f589"
I20260926 07:25:06.411839 2290 raft_consensus.cc:3037] T 776b246a8b4242eda1e6629b8e0414bc P b4e41537987f45039495bb0e26570aea [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:06.411957 2432 raft_consensus.cc:3037] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:06.412806 2290 raft_consensus.cc:2454] T 776b246a8b4242eda1e6629b8e0414bc P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1612b3246374887a6f1e6c837c50f82 in term 1.
I20260926 07:25:06.412962 2432 raft_consensus.cc:2454] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1612b3246374887a6f1e6c837c50f82 in term 1.
I20260926 07:25:06.413128 2504 leader_election.cc:302] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4e41537987f45039495bb0e26570aea, c1612b3246374887a6f1e6c837c50f82; no voters:
I20260926 07:25:06.413301 2714 raft_consensus.cc:2781] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:06.413414 2714 raft_consensus.cc:686] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [term 1 LEADER]: Becoming Leader. State: Replica: c1612b3246374887a6f1e6c837c50f82, State: Running, Role: LEADER
I20260926 07:25:06.413551 2714 consensus_queue.cc:237] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } }
I20260926 07:25:06.415287 2135 catalog_manager.cc:5495] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1612b3246374887a6f1e6c837c50f82 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c1612b3246374887a6f1e6c837c50f82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1612b3246374887a6f1e6c837c50f82" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43847 } health_report { overall_health: HEALTHY } } }
I20260926 07:25:06.435775 2432 raft_consensus.cc:1261] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Refusing update from remote peer c1612b3246374887a6f1e6c837c50f82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:06.435917 2290 raft_consensus.cc:1261] T 776b246a8b4242eda1e6629b8e0414bc P b4e41537987f45039495bb0e26570aea [term 1 FOLLOWER]: Refusing update from remote peer c1612b3246374887a6f1e6c837c50f82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:06.436179 2714 consensus_queue.cc:1035] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f82f3787270748cda591c8ba0476f589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:06.436338 2732 consensus_queue.cc:1035] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4e41537987f45039495bb0e26570aea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:25:06.449411 2719 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 07:25:07.999925 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:08.000070 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:08.002291 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab426e4e6ebb4059a0c8852ffa2e92f3: Sending DeleteTablet for 3 replicas of tablet 776b246a8b4242eda1e6629b8e0414bc
I20260926 07:25:08.002806 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet 776b246a8b4242eda1e6629b8e0414bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42180
I20260926 07:25:08.002871 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 776b246a8b4242eda1e6629b8e0414bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47900
I20260926 07:25:08.003170 2793 tablet_replica.cc:330] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82: stopping tablet replica
I20260926 07:25:08.003141 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 776b246a8b4242eda1e6629b8e0414bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44144
I20260926 07:25:08.003350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 2207
I20260926 07:25:08.004567 2794 tablet_replica.cc:330] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589: stopping tablet replica
I20260926 07:25:08.004714 2794 raft_consensus.cc:2229] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:08.004823 2794 raft_consensus.cc:2258] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:08.004886 2793 raft_consensus.cc:2229] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:25:08.005082 2793 raft_consensus.cc:2258] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:08.005722 2793 ts_tablet_manager.cc:1905] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:08.007404 2794 ts_tablet_manager.cc:1905] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:08.009150 2793 ts_tablet_manager.cc:1918] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:25:08.009231 2793 log.cc:1196] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/776b246a8b4242eda1e6629b8e0414bc
I20260926 07:25:08.009586 2793 ts_tablet_manager.cc:1939] T 776b246a8b4242eda1e6629b8e0414bc P c1612b3246374887a6f1e6c837c50f82: Deleting consensus metadata
I20260926 07:25:08.009656 2794 ts_tablet_manager.cc:1918] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:25:08.009763 2794 log.cc:1196] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/776b246a8b4242eda1e6629b8e0414bc
I20260926 07:25:08.010033 2794 ts_tablet_manager.cc:1939] T 776b246a8b4242eda1e6629b8e0414bc P f82f3787270748cda591c8ba0476f589: Deleting consensus metadata
I20260926 07:25:08.010367 2120 catalog_manager.cc:4841] TS c1612b3246374887a6f1e6c837c50f82 (127.0.98.3:43847): tablet 776b246a8b4242eda1e6629b8e0414bc (table test-table [id=8410d9cd26ac4a6d910415d3dff1a695]) successfully deleted
I20260926 07:25:08.010681 2122 catalog_manager.cc:4841] TS f82f3787270748cda591c8ba0476f589 (127.0.98.2:33295): tablet 776b246a8b4242eda1e6629b8e0414bc (table test-table [id=8410d9cd26ac4a6d910415d3dff1a695]) successfully deleted
I20260926 07:25:08.011588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 2347
W20260926 07:25:08.011607 2120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44567 (error 108)
I20260926 07:25:08.013223 2796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:08.012719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44567 (local address 127.0.0.1:45794)
0926 07:25:08.013007 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:08.013095 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
W20260926 07:25:08.013379 2120 catalog_manager.cc:4584] TS b4e41537987f45039495bb0e26570aea (127.0.98.1:44567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 776b246a8b4242eda1e6629b8e0414bc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44567: connect: Connection refused (error 111)
I20260926 07:25:08.018411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 2487
I20260926 07:25:08.025549 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 2103
2026-09-26T07:25:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6694 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:25:08 dist-test-slave-g04g krb5kdc[2799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:25:08 dist-test-slave-g04g krb5kdc[2799](info): set up 2 sockets
Sep 26 07:25:08 dist-test-slave-g04g krb5kdc[2799](info): commencing operation
krb5kdc: starting...
W20260926 07:25:09.110929 392 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:25:09 dist-test-slave-g04g krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:25:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:25:09.230690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38155
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:09.343792 2812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:09.344075 2812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:09.344166 2812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:09.349272 2812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:09.349367 2812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:09.349404 2812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:09.349445 2812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:09.349479 2812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:09.358404 2812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:09.359968 2812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:09.361096 2812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:09.370280 2817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:09.370335 2818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:09.370441 2812 server_base.cc:1034] running on GCE node
W20260926 07:25:09.370575 2820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:09.371212 2812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:09.372341 2812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:09.373540 2812 hybrid_clock.cc:648] HybridClock initialized: now 1790407509373508 us; error 55 us; skew 500 ppm
Sep 26 07:25:09 dist-test-slave-g04g krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:09.376561 2812 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:25:09.377455 2812 webserver.cc:460] Webserver started at http://127.0.98.62:34835/ using document root <none> and password file <none>
I20260926 07:25:09.378027 2812 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:09.378110 2812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:09.378363 2812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:09.380509 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "6d251757296f47a5955366ec7b52d542"
format_stamp: "Formatted at 2026-09-26 07:25:09 on dist-test-slave-g04g"
server_key: "49b758a44232ea8dead099b56f2b422a"
server_key_iv: "5c91676c033f5179f65f870c834ef354"
server_key_version: "encryptionkey@0"
I20260926 07:25:09.381106 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "6d251757296f47a5955366ec7b52d542"
format_stamp: "Formatted at 2026-09-26 07:25:09 on dist-test-slave-g04g"
server_key: "49b758a44232ea8dead099b56f2b422a"
server_key_iv: "5c91676c033f5179f65f870c834ef354"
server_key_version: "encryptionkey@0"
I20260926 07:25:09.384994 2812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:25:09.387647 2827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:09.388944 2812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:09.389078 2812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "6d251757296f47a5955366ec7b52d542"
format_stamp: "Formatted at 2026-09-26 07:25:09 on dist-test-slave-g04g"
server_key: "49b758a44232ea8dead099b56f2b422a"
server_key_iv: "5c91676c033f5179f65f870c834ef354"
server_key_version: "encryptionkey@0"
I20260926 07:25:09.389200 2812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:09.415928 2812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:09.419394 2812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:09.419600 2812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:09.426658 2812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38155
I20260926 07:25:09.426688 2879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38155 every 8 connection(s)
I20260926 07:25:09.427615 2812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:09.430408 2880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:09.435987 2880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542: Bootstrap starting.
I20260926 07:25:09.436733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 2812
I20260926 07:25:09.436846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:09.437152 392 external_mini_cluster.cc:1428] Setting key 639d728e6818c0a7c0fab39f45016800
I20260926 07:25:09.439121 2880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:09.439868 2880 log.cc:824] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:09.441975 2880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542: No bootstrap required, opened a new log
Sep 26 07:25:09 dist-test-slave-g04g krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:09.446730 2880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d251757296f47a5955366ec7b52d542" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } }
I20260926 07:25:09.446960 2880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:09.447101 2880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d251757296f47a5955366ec7b52d542, State: Initialized, Role: FOLLOWER
I20260926 07:25:09.447635 2880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d251757296f47a5955366ec7b52d542" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } }
I20260926 07:25:09.447854 2880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:09.447942 2880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:09.448060 2880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:09.449378 2880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d251757296f47a5955366ec7b52d542" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } }
I20260926 07:25:09.449759 2880 leader_election.cc:302] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d251757296f47a5955366ec7b52d542; no voters:
I20260926 07:25:09.450104 2880 leader_election.cc:288] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:09.450207 2885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:09.450558 2885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [term 1 LEADER]: Becoming Leader. State: Replica: 6d251757296f47a5955366ec7b52d542, State: Running, Role: LEADER
I20260926 07:25:09.450917 2885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d251757296f47a5955366ec7b52d542" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } }
I20260926 07:25:09.451723 2880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:09.452905 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:09.438489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.0.98.62:38155)
0926 07:25:09.438938 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:09.438953 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:09.438988 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:25:09.439704 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:09.439720 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:09.440083 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:09.440427 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:09.441314 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:09.442342 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:09.443526 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:09.443980 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:09.444101 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:09.447141 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:09.447148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:09.447166 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:09.447203 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:09.449464 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:09.449948 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:09.449954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:09.449961 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:09.450128 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:09.450458 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:09.450463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:09.450468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:09.450993 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:09.451275 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:09.451547 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 07:25:09.451864 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":139,"threads_started":1}
I20260926 07:25:09.452860 2887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d251757296f47a5955366ec7b52d542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d251757296f47a5955366ec7b52d542" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } }
I20260926 07:25:09.452860 2886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d251757296f47a5955366ec7b52d542. Latest consensus state: current_term: 1 leader_uuid: "6d251757296f47a5955366ec7b52d542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d251757296f47a5955366ec7b52d542" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38155 } } }
I20260926 07:25:09.453291 2887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:09.453313 2886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:09.453650 2895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:09.456472 2895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:09.461223 2895 catalog_manager.cc:1347] Generated new cluster ID: 037118c3d4be42a9bbd83a836d32d5b0
I20260926 07:25:09.461290 2895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:09.469660 2895 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:09.470858 2895 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:09.481158 2895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d251757296f47a5955366ec7b52d542: Generated new TSK 0
I20260926 07:25:09.481930 2895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:25:09.514659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38155
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:09.628072 2906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:09.628315 2906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:09.628357 2906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:09.632416 2906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:09.632468 2906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:09.632553 2906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:09.637606 2906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:09.638620 2906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:09.639657 2906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:09.648816 2914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:09.649016 2906 server_base.cc:1034] running on GCE node
W20260926 07:25:09.648816 2912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:09.648904 2911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:09.649521 2906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:09.650110 2906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:09.651298 2906 hybrid_clock.cc:648] HybridClock initialized: now 1790407509651265 us; error 44 us; skew 500 ppm
Sep 26 07:25:09 dist-test-slave-g04g krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:09.654211 2906 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:25:09.655107 2906 webserver.cc:460] Webserver started at http://127.0.98.1:39197/ using document root <none> and password file <none>
I20260926 07:25:09.655613 2906 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:09.655678 2906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:09.655834 2906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:09.657707 2906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "526d8b37ddd64089a1129a1263c8327d"
format_stamp: "Formatted at 2026-09-26 07:25:09 on dist-test-slave-g04g"
server_key: "975163794e5d1c745173aa00b33ceb6c"
server_key_iv: "71cfc57911d912091f4dc9dfe715b00b"
server_key_version: "encryptionkey@0"
I20260926 07:25:09.658193 2906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "526d8b37ddd64089a1129a1263c8327d"
format_stamp: "Formatted at 2026-09-26 07:25:09 on dist-test-slave-g04g"
server_key: "975163794e5d1c745173aa00b33ceb6c"
server_key_iv: "71cfc57911d912091f4dc9dfe715b00b"
server_key_version: "encryptionkey@0"
I20260926 07:25:09.661760 2906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 07:25:09.664206 2921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:09.665504 2906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:09.665642 2906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "526d8b37ddd64089a1129a1263c8327d"
format_stamp: "Formatted at 2026-09-26 07:25:09 on dist-test-slave-g04g"
server_key: "975163794e5d1c745173aa00b33ceb6c"
server_key_iv: "71cfc57911d912091f4dc9dfe715b00b"
server_key_version: "encryptionkey@0"
I20260926 07:25:09.665745 2906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:09.682641 2906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:09.686007 2906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:09.686218 2906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:09.686826 2906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:09.687695 2906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:09.687744 2906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:09.687783 2906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:09.687799 2906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:09.697052 2906 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36557
I20260926 07:25:09.697082 3034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36557 every 8 connection(s)
I20260926 07:25:09.697996 2906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:09.701282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 2906
I20260926 07:25:09.701422 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:25:09.701706 392 external_mini_cluster.cc:1428] Setting key bd7b49536477365e7b59802a9916c146
Sep 26 07:25:09 dist-test-slave-g04g krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 07:25:09.714660 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:09.700036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52383 (local address 127.0.98.62:38155)
0926 07:25:09.700240 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:09.700244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:09.701182 (+ 938us) server_negotiation.cc:407] Connection header received
0926 07:25:09.702958 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:09.702961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:09.703011 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:09.703190 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:09.704969 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:09.705878 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:09.707791 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:09.708049 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:09.708140 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:09.711130 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:09.711133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:09.711135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:09.711159 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:09.712637 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:09.713138 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:09.713142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:09.713144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:09.713221 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:09.713559 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:09.713563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:09.713564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:09.713749 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:09.713860 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:09.714344 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 07:25:09.714460 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 07:25:09.715484 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:09.700479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38155 (local address 127.0.98.1:52383)
0926 07:25:09.701003 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:09.701067 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:09.702705 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:09.703389 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:09.703398 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:09.704001 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:09.704781 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:09.704802 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:09.706030 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:09.706032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:09.707611 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:09.707625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:09.708210 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:09.708214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:09.708456 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:09.709167 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:09.709196 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:09.710911 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:09.712765 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:09.712771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:09.712786 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:09.713034 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:09.713317 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:09.713319 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:09.713321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:09.713456 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:09.713860 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:09.713867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:09.714115 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 07:25:09.714294 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 07:25:09.714624 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":121,"threads_started":1}
I20260926 07:25:09.716619 3035 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38155
I20260926 07:25:09.716884 3035 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:09.717450 3035 heartbeater.cc:511] Master 127.0.98.62:38155 requested a full tablet report, sending...
I20260926 07:25:09.719020 2844 ts_manager.cc:194] Registered new tserver with Master: 526d8b37ddd64089a1129a1263c8327d (127.0.98.1:36557)
I20260926 07:25:09.720173 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:25:09.730657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38155
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:09.841591 3040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:09.841907 3040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:09.841986 3040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:09.846308 3040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:09.846400 3040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:09.846495 3040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:09.851970 3040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:09.853107 3040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:09.854171 3040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:09.863893 3048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:09.863899 3046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:09.864151 3045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:09.864470 3040 server_base.cc:1034] running on GCE node
I20260926 07:25:09.864845 3040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:09.865486 3040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:09.866662 3040 hybrid_clock.cc:648] HybridClock initialized: now 1790407509866606 us; error 67 us; skew 500 ppm
Sep 26 07:25:09 dist-test-slave-g04g krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:09.869630 3040 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:25:09.870488 3040 webserver.cc:460] Webserver started at http://127.0.98.2:37707/ using document root <none> and password file <none>
I20260926 07:25:09.871086 3040 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:09.871209 3040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:09.871416 3040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:09.873366 3040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "f82154a396dd4b059cdf7a2eb0844c7f"
format_stamp: "Formatted at 2026-09-26 07:25:09 on dist-test-slave-g04g"
server_key: "c81e657a89d3fc6f227af8516cae84ea"
server_key_iv: "e3fd9ac1848ace0158aa491269d79e06"
server_key_version: "encryptionkey@0"
I20260926 07:25:09.873910 3040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f82154a396dd4b059cdf7a2eb0844c7f"
format_stamp: "Formatted at 2026-09-26 07:25:09 on dist-test-slave-g04g"
server_key: "c81e657a89d3fc6f227af8516cae84ea"
server_key_iv: "e3fd9ac1848ace0158aa491269d79e06"
server_key_version: "encryptionkey@0"
I20260926 07:25:09.877547 3040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:25:09.879813 3055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:09.880975 3040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:09.881100 3040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "f82154a396dd4b059cdf7a2eb0844c7f"
format_stamp: "Formatted at 2026-09-26 07:25:09 on dist-test-slave-g04g"
server_key: "c81e657a89d3fc6f227af8516cae84ea"
server_key_iv: "e3fd9ac1848ace0158aa491269d79e06"
server_key_version: "encryptionkey@0"
I20260926 07:25:09.881230 3040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:09.906844 3040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:09.910089 3040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:09.910291 3040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:09.910856 3040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:09.911693 3040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:09.911757 3040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:09.911824 3040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:09.911870 3040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:09.920667 3040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40547
I20260926 07:25:09.920696 3168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40547 every 8 connection(s)
I20260926 07:25:09.921500 3040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:25:09.926785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 3040
I20260926 07:25:09.926904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:25:09.927243 392 external_mini_cluster.cc:1428] Setting key e2344f50a3f9d6450850d27b4684aec0
Sep 26 07:25:09 dist-test-slave-g04g krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:09.936106 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:09.923464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50433 (local address 127.0.98.62:38155)
0926 07:25:09.923595 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:09.923600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:09.924448 (+ 848us) server_negotiation.cc:407] Connection header received
0926 07:25:09.925735 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:09.925738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:09.925783 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:09.925858 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:09.927101 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:09.927783 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:09.929033 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:09.929258 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:09.929362 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:09.932265 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:09.932269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:09.932272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:09.932303 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:09.933822 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:09.934460 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:09.934462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:09.934463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:09.934533 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:09.934926 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:09.934929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:09.934930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:09.935173 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:09.935253 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:09.935895 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 07:25:09.935989 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 07:25:09.937306 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:09.923906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38155 (local address 127.0.98.2:50433)
0926 07:25:09.924312 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:09.924351 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:09.925546 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:09.926006 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:09.926012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:09.926372 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:09.926857 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:09.926871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:09.927908 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:09.927912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:09.928912 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:09.928922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:09.929395 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:09.929399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:09.929561 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:09.930180 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:09.930204 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:09.932100 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:09.933936 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:09.933946 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:09.933964 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:09.934351 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:09.934626 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:09.934630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:09.934632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:09.934825 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:09.935274 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:09.935283 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:09.935626 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 07:25:09.935845 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 07:25:09.936205 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260926 07:25:09.938448 3169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38155
I20260926 07:25:09.938681 3169 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:09.939260 3169 heartbeater.cc:511] Master 127.0.98.62:38155 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 07:25:09.940342 2844 ts_manager.cc:194] Registered new tserver with Master: f82154a396dd4b059cdf7a2eb0844c7f (127.0.98.2:40547)
I20260926 07:25:09.940991 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:25:09.954653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38155
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:10.066830 3174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:10.067121 3174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:10.067174 3174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:10.071319 3174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:10.071368 3174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:10.071424 3174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:10.076524 3174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:10.077466 3174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:10.078413 3174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:25:10.087857 3174 server_base.cc:1034] running on GCE node
W20260926 07:25:10.087787 3179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:10.087796 3180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:10.087787 3182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:10.088454 3174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:10.089087 3174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:10.090281 3174 hybrid_clock.cc:648] HybridClock initialized: now 1790407510090268 us; error 26 us; skew 500 ppm
Sep 26 07:25:10 dist-test-slave-g04g krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:10.093245 3174 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:25:10.094162 3174 webserver.cc:460] Webserver started at http://127.0.98.3:38391/ using document root <none> and password file <none>
I20260926 07:25:10.094704 3174 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:10.094787 3174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:10.094992 3174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:10.096948 3174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "e45c5d2991cf4ede899274847920073b"
format_stamp: "Formatted at 2026-09-26 07:25:10 on dist-test-slave-g04g"
server_key: "db2c03f50939c4c8c49ced6dc48eb4b2"
server_key_iv: "d3694c3525f8b74606c825e5a284a38d"
server_key_version: "encryptionkey@0"
I20260926 07:25:10.097487 3174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e45c5d2991cf4ede899274847920073b"
format_stamp: "Formatted at 2026-09-26 07:25:10 on dist-test-slave-g04g"
server_key: "db2c03f50939c4c8c49ced6dc48eb4b2"
server_key_iv: "d3694c3525f8b74606c825e5a284a38d"
server_key_version: "encryptionkey@0"
I20260926 07:25:10.101020 3174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 07:25:10.103304 3189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:10.104450 3174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 07:25:10.104573 3174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "e45c5d2991cf4ede899274847920073b"
format_stamp: "Formatted at 2026-09-26 07:25:10 on dist-test-slave-g04g"
server_key: "db2c03f50939c4c8c49ced6dc48eb4b2"
server_key_iv: "d3694c3525f8b74606c825e5a284a38d"
server_key_version: "encryptionkey@0"
I20260926 07:25:10.104676 3174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:10.114039 3174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:10.117084 3174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:10.117268 3174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:10.117790 3174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:10.118541 3174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:10.118604 3174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:10.118666 3174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:10.118706 3174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:10.127372 3174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44169
I20260926 07:25:10.127396 3302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44169 every 8 connection(s)
I20260926 07:25:10.128206 3174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:10.130255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 3174
I20260926 07:25:10.130368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:10.130820 392 external_mini_cluster.cc:1428] Setting key f10629df2313eee2eeb6c747eea49e98
Sep 26 07:25:10 dist-test-slave-g04g krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:10.141227 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:10.129856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60635 (local address 127.0.98.62:38155)
0926 07:25:10.129986 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:10.129990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:10.131027 (+ 1037us) server_negotiation.cc:407] Connection header received
0926 07:25:10.132735 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:10.132738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:10.132814 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:10.132895 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:10.133957 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:10.134495 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:10.135646 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:10.135849 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:10.135926 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:10.138156 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:10.138159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:10.138161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:10.138183 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:10.139572 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:10.139985 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:10.139987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:10.139989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:10.140053 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:10.140332 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:10.140334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:10.140335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:10.140483 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:10.140553 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:10.141008 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 07:25:10.141088 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:25:10.141999 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:10.130332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38155 (local address 127.0.98.3:60635)
0926 07:25:10.130862 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:10.130928 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:10.132527 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:10.133062 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:10.133068 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:10.133395 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:10.133835 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:10.133847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:10.134594 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:10.134596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:10.135545 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:10.135554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:10.135961 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:10.135963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:10.136092 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:10.136608 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:10.136625 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:10.137978 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:10.139659 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:10.139663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:10.139674 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:10.139909 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:10.140146 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:10.140148 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:10.140150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:10.140261 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:10.140588 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:10.140593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:10.140815 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 07:25:10.140971 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 07:25:10.141246 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"spinlock_wait_cycles":56064,"thread_start_us":135,"threads_started":1}
I20260926 07:25:10.142933 3303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38155
I20260926 07:25:10.143255 3303 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:10.143771 3303 heartbeater.cc:511] Master 127.0.98.62:38155 requested a full tablet report, sending...
I20260926 07:25:10.144701 2844 ts_manager.cc:194] Registered new tserver with Master: e45c5d2991cf4ede899274847920073b (127.0.98.3:44169)
I20260926 07:25:10.145224 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60635
I20260926 07:25:10.155787 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:25:10.167659 2883 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:10.166344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.0.98.62:38155)
0926 07:25:10.166454 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:10.166457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:10.166647 (+ 190us) server_negotiation.cc:407] Connection header received
0926 07:25:10.166747 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:10.166749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:10.166786 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:10.166855 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:10.167559 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34420: BlockingRecv error: recv got EOF from 127.0.0.1:34420 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:25:10.168684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 2906
I20260926 07:25:10.174161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 3040
I20260926 07:25:10.179361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 3174
I20260926 07:25:10.184480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 2812
2026-09-26T07:25:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2156 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:25:10 dist-test-slave-g04g krb5kdc[3315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:25:10 dist-test-slave-g04g krb5kdc[3315](info): set up 2 sockets
Sep 26 07:25:10 dist-test-slave-g04g krb5kdc[3315](info): commencing operation
krb5kdc: starting...
W20260926 07:25:11.270974 392 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:25:11 dist-test-slave-g04g krb5kdc[3315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:25:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:25:11.390995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36321
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:11.498930 3328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:11.499233 3328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:11.499315 3328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:11.503245 3328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:11.503322 3328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:11.503360 3328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:11.503393 3328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:11.503424 3328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:11.508453 3328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:11.509522 3328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:11.510516 3328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:11.518705 3333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:11.518738 3334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:11.518824 3328 server_base.cc:1034] running on GCE node
W20260926 07:25:11.518725 3336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:11.519508 3328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:11.520547 3328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:11.521709 3328 hybrid_clock.cc:648] HybridClock initialized: now 1790407511521694 us; error 26 us; skew 500 ppm
Sep 26 07:25:11 dist-test-slave-g04g krb5kdc[3315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:11.528674 3328 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:25:11.529475 3328 webserver.cc:460] Webserver started at http://127.0.98.62:38907/ using document root <none> and password file <none>
I20260926 07:25:11.529982 3328 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:11.530040 3328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:11.530197 3328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:11.532231 3328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "94807fdf8ef246e982c1328ba4e606dd"
format_stamp: "Formatted at 2026-09-26 07:25:11 on dist-test-slave-g04g"
server_key: "2ff459dfa61ee6869de9d55244f57e34"
server_key_iv: "90942550a0b544bbc2730b87908306d3"
server_key_version: "encryptionkey@0"
I20260926 07:25:11.532732 3328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "94807fdf8ef246e982c1328ba4e606dd"
format_stamp: "Formatted at 2026-09-26 07:25:11 on dist-test-slave-g04g"
server_key: "2ff459dfa61ee6869de9d55244f57e34"
server_key_iv: "90942550a0b544bbc2730b87908306d3"
server_key_version: "encryptionkey@0"
I20260926 07:25:11.536345 3328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:11.538709 3343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:11.540067 3328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:11.540215 3328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "94807fdf8ef246e982c1328ba4e606dd"
format_stamp: "Formatted at 2026-09-26 07:25:11 on dist-test-slave-g04g"
server_key: "2ff459dfa61ee6869de9d55244f57e34"
server_key_iv: "90942550a0b544bbc2730b87908306d3"
server_key_version: "encryptionkey@0"
I20260926 07:25:11.540321 3328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:11.568272 3328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:11.571450 3328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:11.571619 3328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:11.579144 3328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36321
I20260926 07:25:11.579424 3395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36321 every 8 connection(s)
I20260926 07:25:11.580193 3328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:11.582967 3396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:11.586951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 3328
I20260926 07:25:11.587092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:11.587383 392 external_mini_cluster.cc:1428] Setting key 05de73f58c34ccacb7c3ff786edf541e
I20260926 07:25:11.589488 3396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd: Bootstrap starting.
I20260926 07:25:11.591703 3396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:11.592422 3396 log.cc:824] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:11.594180 3396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd: No bootstrap required, opened a new log
Sep 26 07:25:11 dist-test-slave-g04g krb5kdc[3315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:11.598147 3396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94807fdf8ef246e982c1328ba4e606dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36321 } }
I20260926 07:25:11.598388 3396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:11.598511 3396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94807fdf8ef246e982c1328ba4e606dd, State: Initialized, Role: FOLLOWER
I20260926 07:25:11.599010 3396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94807fdf8ef246e982c1328ba4e606dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36321 } }
I20260926 07:25:11.599211 3396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:11.599282 3396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:11.599400 3396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:11.600518 3396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94807fdf8ef246e982c1328ba4e606dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36321 } }
I20260926 07:25:11.600850 3396 leader_election.cc:302] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94807fdf8ef246e982c1328ba4e606dd; no voters:
I20260926 07:25:11.601157 3396 leader_election.cc:288] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:11.601279 3401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:11.601500 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:11.588823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.0.98.62:36321)
0926 07:25:11.589266 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:11.589281 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:11.589316 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:25:11.590004 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:11.590022 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:11.590365 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:11.590697 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:11.591512 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:11.592561 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:11.593401 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:11.593887 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:11.594070 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:11.596664 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:11.596669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:11.596683 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:11.596709 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:11.598365 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:11.598880 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:11.598885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:11.598889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:11.599001 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:11.599363 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:11.599369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:11.599372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:11.599811 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:11.599970 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:11.600225 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 07:25:11.600532 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":121,"threads_started":1}
I20260926 07:25:11.601495 3401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [term 1 LEADER]: Becoming Leader. State: Replica: 94807fdf8ef246e982c1328ba4e606dd, State: Running, Role: LEADER
I20260926 07:25:11.602239 3401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94807fdf8ef246e982c1328ba4e606dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36321 } }
I20260926 07:25:11.602401 3396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:11.604447 3403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94807fdf8ef246e982c1328ba4e606dd. Latest consensus state: current_term: 1 leader_uuid: "94807fdf8ef246e982c1328ba4e606dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94807fdf8ef246e982c1328ba4e606dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36321 } } }
I20260926 07:25:11.604583 3403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:11.604862 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:25:11.604894 3402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94807fdf8ef246e982c1328ba4e606dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94807fdf8ef246e982c1328ba4e606dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36321 } } }
I20260926 07:25:11.607168 3402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd [sys.catalog]: This master's current role is: LEADER
W20260926 07:25:11.607594 3415 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:25:11.607656 3415 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:25:11.607734 3416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:11.608742 3416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:11.615850 3416 catalog_manager.cc:1347] Generated new cluster ID: 3dc36ad8712d4023aef77468b3e2068e
I20260926 07:25:11.615918 3416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:11.624019 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:11.619282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.0.98.62:36321)
0926 07:25:11.619657 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:11.619662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:11.619677 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:25:11.619803 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:11.619806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:11.619855 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:11.619947 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:11.620472 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:11.621043 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:11.621867 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:11.622119 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:11.622224 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:11.623099 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:11.623103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:11.623105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:11.623569 (+ 464us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:11.623597 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:11.623613 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:11.623711 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:11.623809 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 07:25:11.623868 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282}
I20260926 07:25:11.637377 3416 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:11.638430 3416 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:11.649601 3416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94807fdf8ef246e982c1328ba4e606dd: Generated new TSK 0
I20260926 07:25:11.650457 3416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:25:11.663875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 3328
2026-09-26T07:25:11Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1477 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:25:11 dist-test-slave-g04g krb5kdc[3429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:25:11 dist-test-slave-g04g krb5kdc[3429](info): set up 2 sockets
Sep 26 07:25:11 dist-test-slave-g04g krb5kdc[3429](info): commencing operation
krb5kdc: starting...
W20260926 07:25:12.699189 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:25:12 dist-test-slave-g04g krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:25:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:25:12.819702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36743
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:12.933562 3442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:12.933826 3442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:12.933867 3442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:12.937742 3442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:12.937790 3442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:12.937809 3442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:12.937826 3442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:12.937840 3442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:12.942919 3442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:12.943964 3442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:12.944976 3442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:12.953703 3448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:12.953679 3450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:12.953697 3442 server_base.cc:1034] running on GCE node
W20260926 07:25:12.953740 3447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:12.954536 3442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:12.955681 3442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:12.956843 3442 hybrid_clock.cc:648] HybridClock initialized: now 1790407512956828 us; error 29 us; skew 500 ppm
Sep 26 07:25:12 dist-test-slave-g04g krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:12.959650 3442 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:25:12.960537 3442 webserver.cc:460] Webserver started at http://127.0.98.62:42455/ using document root <none> and password file <none>
I20260926 07:25:12.961082 3442 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:12.961163 3442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:12.961387 3442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:12.963435 3442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "8e3635d53cd14daeb12f48e16e7d91c3"
format_stamp: "Formatted at 2026-09-26 07:25:12 on dist-test-slave-g04g"
server_key: "0735e6e99690408c2ded6a1392aa4296"
server_key_iv: "cde893bd44c92eebdeb947e4cf0bcc8b"
server_key_version: "encryptionkey@0"
I20260926 07:25:12.963985 3442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e3635d53cd14daeb12f48e16e7d91c3"
format_stamp: "Formatted at 2026-09-26 07:25:12 on dist-test-slave-g04g"
server_key: "0735e6e99690408c2ded6a1392aa4296"
server_key_iv: "cde893bd44c92eebdeb947e4cf0bcc8b"
server_key_version: "encryptionkey@0"
I20260926 07:25:12.967540 3442 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:12.969744 3457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:12.970950 3442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:12.971094 3442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "8e3635d53cd14daeb12f48e16e7d91c3"
format_stamp: "Formatted at 2026-09-26 07:25:12 on dist-test-slave-g04g"
server_key: "0735e6e99690408c2ded6a1392aa4296"
server_key_iv: "cde893bd44c92eebdeb947e4cf0bcc8b"
server_key_version: "encryptionkey@0"
I20260926 07:25:12.971196 3442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:12.993674 3442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:12.996445 3442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:12.996639 3442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:13.003114 3442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36743
I20260926 07:25:13.003441 3509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36743 every 8 connection(s)
I20260926 07:25:13.004040 3442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:13.006737 3510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:13.011293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 3442
I20260926 07:25:13.011412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:13.011664 392 external_mini_cluster.cc:1428] Setting key 2d1fccc3bcba6aa607c74039b88068bc
I20260926 07:25:13.012971 3510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3: Bootstrap starting.
I20260926 07:25:13.015309 3510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:13.016105 3510 log.cc:824] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:13.018141 3510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3: No bootstrap required, opened a new log
Sep 26 07:25:13 dist-test-slave-g04g krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:13.022168 3510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3635d53cd14daeb12f48e16e7d91c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } }
I20260926 07:25:13.022393 3510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:13.022455 3510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e3635d53cd14daeb12f48e16e7d91c3, State: Initialized, Role: FOLLOWER
I20260926 07:25:13.022909 3510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3635d53cd14daeb12f48e16e7d91c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } }
I20260926 07:25:13.023064 3510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:13.023132 3510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:13.023231 3510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:13.024056 3510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3635d53cd14daeb12f48e16e7d91c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } }
I20260926 07:25:13.024387 3510 leader_election.cc:302] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e3635d53cd14daeb12f48e16e7d91c3; no voters:
I20260926 07:25:13.024698 3510 leader_election.cc:288] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:13.024755 3515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:13.024971 3515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [term 1 LEADER]: Becoming Leader. State: Replica: 8e3635d53cd14daeb12f48e16e7d91c3, State: Running, Role: LEADER
I20260926 07:25:13.025457 3515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3635d53cd14daeb12f48e16e7d91c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } }
I20260926 07:25:13.025991 3510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:13.026419 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:13.013095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.0.98.62:36743)
0926 07:25:13.013495 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:13.013510 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:13.013545 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:25:13.014158 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:13.014174 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:13.014501 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:13.014769 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:13.015735 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.016748 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.017857 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.018364 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.018482 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:13.021260 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:13.021265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:13.021283 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:13.021309 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:13.023243 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:13.023714 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:13.023720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:13.023726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:13.023843 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:13.024110 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:13.024114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:13.024116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:13.024549 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:13.024746 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:13.025318 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 07:25:13.025602 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":128,"threads_started":1}
I20260926 07:25:13.026609 3515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e3635d53cd14daeb12f48e16e7d91c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3635d53cd14daeb12f48e16e7d91c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } } }
I20260926 07:25:13.026743 3515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:13.027019 3516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e3635d53cd14daeb12f48e16e7d91c3. Latest consensus state: current_term: 1 leader_uuid: "8e3635d53cd14daeb12f48e16e7d91c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3635d53cd14daeb12f48e16e7d91c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } } }
I20260926 07:25:13.027138 3516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3 [sys.catalog]: This master's current role is: LEADER
W20260926 07:25:13.030766 3530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:25:13.030853 3530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:25:13.030910 3522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:13.031736 3522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:13.037401 3522 catalog_manager.cc:1347] Generated new cluster ID: b8fb8c8cfa79407990e4642572b66b8b
I20260926 07:25:13.037477 3522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:13.071059 3522 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:13.072017 3522 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:13.078420 3522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e3635d53cd14daeb12f48e16e7d91c3: Generated new TSK 0
I20260926 07:25:13.079118 3522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:25:13.114444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36743
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:13.227754 3536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:13.228051 3536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:13.228122 3536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:13.232206 3536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:13.232264 3536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:13.232357 3536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:13.237466 3536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:13.238494 3536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:13.239589 3536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:13.248950 3541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:13.248997 3542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:13.249155 3544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:13.249258 3536 server_base.cc:1034] running on GCE node
I20260926 07:25:13.249734 3536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:13.250357 3536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:13.251528 3536 hybrid_clock.cc:648] HybridClock initialized: now 1790407513251511 us; error 29 us; skew 500 ppm
Sep 26 07:25:13 dist-test-slave-g04g krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:13.254297 3536 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:25:13.255172 3536 webserver.cc:460] Webserver started at http://127.0.98.1:37517/ using document root <none> and password file <none>
I20260926 07:25:13.255719 3536 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:13.255803 3536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:13.256027 3536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:13.257982 3536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "2708bc92f54f434f8dc1f25042de9078"
format_stamp: "Formatted at 2026-09-26 07:25:13 on dist-test-slave-g04g"
server_key: "d935f136aafa42ff32e8ecb478cadeed"
server_key_iv: "4bcfbafc62a86b415a9be6d7f62eef2e"
server_key_version: "encryptionkey@0"
I20260926 07:25:13.258524 3536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2708bc92f54f434f8dc1f25042de9078"
format_stamp: "Formatted at 2026-09-26 07:25:13 on dist-test-slave-g04g"
server_key: "d935f136aafa42ff32e8ecb478cadeed"
server_key_iv: "4bcfbafc62a86b415a9be6d7f62eef2e"
server_key_version: "encryptionkey@0"
I20260926 07:25:13.262076 3536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 07:25:13.264410 3551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:13.265575 3536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:13.265703 3536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "2708bc92f54f434f8dc1f25042de9078"
format_stamp: "Formatted at 2026-09-26 07:25:13 on dist-test-slave-g04g"
server_key: "d935f136aafa42ff32e8ecb478cadeed"
server_key_iv: "4bcfbafc62a86b415a9be6d7f62eef2e"
server_key_version: "encryptionkey@0"
I20260926 07:25:13.265802 3536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:13.288848 3536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:13.292182 3536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:13.292387 3536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:13.292979 3536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:13.293783 3536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:13.293853 3536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:13.293919 3536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:13.293964 3536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:13.303146 3536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45851
I20260926 07:25:13.303177 3664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45851 every 8 connection(s)
I20260926 07:25:13.304067 3536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:13.310606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 3536
I20260926 07:25:13.310743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:25:13.311146 392 external_mini_cluster.cc:1428] Setting key f31fdb1c80d068d518c2c69e52e0f4c7
Sep 26 07:25:13 dist-test-slave-g04g krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:13.319971 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:13.306038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37695 (local address 127.0.98.62:36743)
0926 07:25:13.306198 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:13.306202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:13.307049 (+ 847us) server_negotiation.cc:407] Connection header received
0926 07:25:13.308562 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:13.308565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:13.308625 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:13.308723 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:13.309893 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.310444 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.312082 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.312402 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.312515 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:13.315357 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:13.315363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:13.315368 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:13.315405 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:13.317245 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:13.318010 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:13.318047 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:13.318050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:13.318181 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:13.318715 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:13.318718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:13.318719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:13.318942 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:13.319025 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:13.319713 (+ 688us) server_negotiation.cc:299] Negotiation successful
0926 07:25:13.319819 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:25:13.321197 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:13.306429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36743 (local address 127.0.98.1:37695)
0926 07:25:13.306868 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:13.306913 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:13.308316 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:13.308875 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:13.308882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:13.309231 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:13.309728 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:13.309741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.310594 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.310599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:13.311959 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:13.311968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.312576 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.312580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:13.312780 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:13.313425 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:13.313449 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:13.315174 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:13.317455 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:13.317462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:13.317482 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:13.317878 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:13.318364 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:13.318389 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:13.318393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:13.318596 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:13.319076 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:13.319084 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:13.319433 (+ 349us) client_negotiation.cc:769] Sending connection context
0926 07:25:13.319649 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 07:25:13.320053 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 07:25:13.322484 3665 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36743
I20260926 07:25:13.322739 3665 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:13.323272 3665 heartbeater.cc:511] Master 127.0.98.62:36743 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 07:25:13.324852 3474 ts_manager.cc:194] Registered new tserver with Master: 2708bc92f54f434f8dc1f25042de9078 (127.0.98.1:45851)
I20260926 07:25:13.325992 3474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:25:13.338632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36743
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:13.445832 3670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:13.446069 3670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:13.446132 3670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:13.450270 3670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:13.450330 3670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:13.450439 3670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:13.455639 3670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:13.456634 3670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:13.457661 3670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:13.467296 3675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:13.467305 3678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:13.467453 3670 server_base.cc:1034] running on GCE node
W20260926 07:25:13.467309 3676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:13.467875 3670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:13.468523 3670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:13.469698 3670 hybrid_clock.cc:648] HybridClock initialized: now 1790407513469668 us; error 46 us; skew 500 ppm
Sep 26 07:25:13 dist-test-slave-g04g krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:13.472513 3670 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:25:13.473359 3670 webserver.cc:460] Webserver started at http://127.0.98.2:42329/ using document root <none> and password file <none>
I20260926 07:25:13.474007 3670 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:13.474109 3670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:13.474340 3670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:13.476450 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "40b147ec26914d07bfb2f47ccc5c5bce"
format_stamp: "Formatted at 2026-09-26 07:25:13 on dist-test-slave-g04g"
server_key: "4a9b0b2817a6b619c2524436781f5d3e"
server_key_iv: "5c712de4e216c856afc0571c15641254"
server_key_version: "encryptionkey@0"
I20260926 07:25:13.477015 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "40b147ec26914d07bfb2f47ccc5c5bce"
format_stamp: "Formatted at 2026-09-26 07:25:13 on dist-test-slave-g04g"
server_key: "4a9b0b2817a6b619c2524436781f5d3e"
server_key_iv: "5c712de4e216c856afc0571c15641254"
server_key_version: "encryptionkey@0"
I20260926 07:25:13.480916 3670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:25:13.483546 3685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:13.484745 3670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:13.484902 3670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "40b147ec26914d07bfb2f47ccc5c5bce"
format_stamp: "Formatted at 2026-09-26 07:25:13 on dist-test-slave-g04g"
server_key: "4a9b0b2817a6b619c2524436781f5d3e"
server_key_iv: "5c712de4e216c856afc0571c15641254"
server_key_version: "encryptionkey@0"
I20260926 07:25:13.485013 3670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:13.502209 3670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:13.505393 3670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:13.505596 3670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:13.506178 3670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:13.506968 3670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:13.507032 3670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:13.507150 3670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:13.507196 3670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:13.516041 3670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32947
I20260926 07:25:13.516062 3798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32947 every 8 connection(s)
I20260926 07:25:13.516913 3670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:25:13.524559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 3670
I20260926 07:25:13.524657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:25:13.524981 392 external_mini_cluster.cc:1428] Setting key 60b121023d8c9c33e8786e1c52357714
Sep 26 07:25:13 dist-test-slave-g04g krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:13.530995 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:13.518806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42077 (local address 127.0.98.62:36743)
0926 07:25:13.518939 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:13.518944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:13.519642 (+ 698us) server_negotiation.cc:407] Connection header received
0926 07:25:13.521038 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:13.521041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:13.521100 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:13.521179 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:13.522304 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.522844 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.524014 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.524250 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.524399 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:13.527387 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:13.527390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:13.527393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:13.527417 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:13.529046 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:13.529537 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:13.529541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:13.529542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:13.529620 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:13.529957 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:13.529960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:13.529962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:13.530137 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:13.530233 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:13.530714 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 07:25:13.530833 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:25:13.531839 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:13.519088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36743 (local address 127.0.98.2:42077)
0926 07:25:13.519492 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:13.519532 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:13.520784 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:13.521339 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:13.521347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:13.521686 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:13.522154 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:13.522166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.522980 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.522983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:13.523888 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:13.523897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.524448 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.524451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:13.524596 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:13.525321 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:13.525343 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:13.527238 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:13.529152 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:13.529158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:13.529171 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:13.529434 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:13.529718 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:13.529721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:13.529723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:13.529862 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:13.530250 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:13.530256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:13.530486 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 07:25:13.530659 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 07:25:13.530977 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":97,"threads_started":1}
I20260926 07:25:13.532939 3799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36743
I20260926 07:25:13.533212 3799 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:13.533803 3799 heartbeater.cc:511] Master 127.0.98.62:36743 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 07:25:13.534834 3474 ts_manager.cc:194] Registered new tserver with Master: 40b147ec26914d07bfb2f47ccc5c5bce (127.0.98.2:32947)
I20260926 07:25:13.535471 3474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:25:13.550645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36743
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:13.667896 3804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:13.668159 3804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:13.668208 3804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:13.672806 3804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:13.672878 3804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:13.672945 3804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:13.678344 3804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:13.679276 3804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:13.680195 3804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:13.689594 3809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:13.689585 3812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:13.689610 3810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:13.689806 3804 server_base.cc:1034] running on GCE node
I20260926 07:25:13.690246 3804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:13.690835 3804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:13.692004 3804 hybrid_clock.cc:648] HybridClock initialized: now 1790407513691985 us; error 60 us; skew 500 ppm
Sep 26 07:25:13 dist-test-slave-g04g krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:13.694679 3804 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:25:13.695557 3804 webserver.cc:460] Webserver started at http://127.0.98.3:40947/ using document root <none> and password file <none>
I20260926 07:25:13.696087 3804 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:13.696174 3804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:13.696388 3804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:13.698346 3804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "63ff5ee1d52e4554acd0bdee9b1368e7"
format_stamp: "Formatted at 2026-09-26 07:25:13 on dist-test-slave-g04g"
server_key: "15515b5de27befcd10592fc3187249b8"
server_key_iv: "6de9923683fbeb18a142895bd7528abb"
server_key_version: "encryptionkey@0"
I20260926 07:25:13.698876 3804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "63ff5ee1d52e4554acd0bdee9b1368e7"
format_stamp: "Formatted at 2026-09-26 07:25:13 on dist-test-slave-g04g"
server_key: "15515b5de27befcd10592fc3187249b8"
server_key_iv: "6de9923683fbeb18a142895bd7528abb"
server_key_version: "encryptionkey@0"
I20260926 07:25:13.702497 3804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:25:13.704973 3819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:13.706215 3804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:13.706352 3804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "63ff5ee1d52e4554acd0bdee9b1368e7"
format_stamp: "Formatted at 2026-09-26 07:25:13 on dist-test-slave-g04g"
server_key: "15515b5de27befcd10592fc3187249b8"
server_key_iv: "6de9923683fbeb18a142895bd7528abb"
server_key_version: "encryptionkey@0"
I20260926 07:25:13.706456 3804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:13.721397 3804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:13.724479 3804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:13.724666 3804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:13.725204 3804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:13.726002 3804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:13.726068 3804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:13.726131 3804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:13.726178 3804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:13.734725 3804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40499
I20260926 07:25:13.734750 3932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40499 every 8 connection(s)
I20260926 07:25:13.735739 3804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:13.736580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 3804
I20260926 07:25:13.736691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:13.736979 392 external_mini_cluster.cc:1428] Setting key 3f7b7177c851c5e73a7305e932586392
Sep 26 07:25:13 dist-test-slave-g04g krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:13.748843 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:13.737701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45621 (local address 127.0.98.62:36743)
0926 07:25:13.737930 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:13.737934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:13.738662 (+ 728us) server_negotiation.cc:407] Connection header received
0926 07:25:13.740123 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:13.740126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:13.740169 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:13.740239 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:13.741331 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.741841 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.742911 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.743135 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.743207 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:13.745623 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:13.745626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:13.745628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:13.745651 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:13.747083 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:13.747523 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:13.747525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:13.747527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:13.747599 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:13.747948 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:13.747950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:13.747952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:13.748113 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:13.748185 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:13.748629 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 07:25:13.748720 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260926 07:25:13.749971 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:13.737914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36743 (local address 127.0.98.3:45621)
0926 07:25:13.738504 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:13.738558 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:13.739939 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:13.740407 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:13.740413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:13.740746 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:13.741208 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:13.741220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.741941 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.741944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:13.742813 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:13.742822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.743258 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.743261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:13.743416 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:13.744063 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:13.744078 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:13.745476 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:13.747184 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:13.747189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:13.747199 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:13.747440 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:13.747706 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:13.747710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:13.747712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:13.747859 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:13.748196 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:13.748207 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:13.748438 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 07:25:13.748591 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 07:25:13.748846 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":117,"threads_started":1}
I20260926 07:25:13.751102 3933 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36743
I20260926 07:25:13.751343 3933 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:13.751798 3933 heartbeater.cc:511] Master 127.0.98.62:36743 requested a full tablet report, sending...
I20260926 07:25:13.752792 3473 ts_manager.cc:194] Registered new tserver with Master: 63ff5ee1d52e4554acd0bdee9b1368e7 (127.0.98.3:40499)
I20260926 07:25:13.753357 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45621
I20260926 07:25:13.762645 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:25:13 dist-test-slave-g04g krb5kdc[3429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407513, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 07:25:13 dist-test-slave-g04g krb5kdc[3429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407513, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:13.786846 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:13.778946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54966 (local address 127.0.98.62:36743)
0926 07:25:13.779107 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:13.779112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:13.779232 (+ 120us) server_negotiation.cc:407] Connection header received
0926 07:25:13.779344 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:13.779347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:13.779388 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:13.779457 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:13.780070 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.780576 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.781333 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:13.781537 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:13.781620 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:13.783959 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:13.783962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:13.783964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:13.783985 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:13.785385 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:13.785769 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:13.785772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:13.785773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:13.785844 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:13.786053 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:13.786055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:13.786056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:13.786253 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:13.786351 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:13.786514 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 07:25:13.786595 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 07:25:13.787249 3473 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54966
I20260926 07:25:13.787997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 3536
I20260926 07:25:13.793915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 3670
I20260926 07:25:13.799099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 3804
I20260926 07:25:13.804209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 3442
2026-09-26T07:25:13Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2142 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:25:13 dist-test-slave-g04g krb5kdc[3945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:25:13 dist-test-slave-g04g krb5kdc[3945](info): set up 2 sockets
Sep 26 07:25:13 dist-test-slave-g04g krb5kdc[3945](info): commencing operation
krb5kdc: starting...
W20260926 07:25:14.839103 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:25:14 dist-test-slave-g04g krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:25:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:25:14.962985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:15.072347 3958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:15.072634 3958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:15.072721 3958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:15.076840 3958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:15.076915 3958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:15.076952 3958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:15.077003 3958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:15.077040 3958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:15.082242 3958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:15.083252 3958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:15.084240 3958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:15.092495 3964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:15.092530 3963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:15.092746 3958 server_base.cc:1034] running on GCE node
W20260926 07:25:15.092509 3966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:15.093396 3958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:15.094515 3958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:15.095700 3958 hybrid_clock.cc:648] HybridClock initialized: now 1790407515095677 us; error 35 us; skew 500 ppm
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:15.098436 3958 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:25:15.099308 3958 webserver.cc:460] Webserver started at http://127.0.98.62:35523/ using document root <none> and password file <none>
I20260926 07:25:15.099819 3958 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:15.099877 3958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:15.100033 3958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:15.101982 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "0cb7295bbc7a4c9c94a9486d2ec497e2"
format_stamp: "Formatted at 2026-09-26 07:25:15 on dist-test-slave-g04g"
server_key: "f95d491e7e3e597233c4afd90963cdfb"
server_key_iv: "aed51bc121201bb177d39e56d6ae3c05"
server_key_version: "encryptionkey@0"
I20260926 07:25:15.102483 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "0cb7295bbc7a4c9c94a9486d2ec497e2"
format_stamp: "Formatted at 2026-09-26 07:25:15 on dist-test-slave-g04g"
server_key: "f95d491e7e3e597233c4afd90963cdfb"
server_key_iv: "aed51bc121201bb177d39e56d6ae3c05"
server_key_version: "encryptionkey@0"
I20260926 07:25:15.106098 3958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:25:15.108538 3973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:15.109884 3958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:15.109998 3958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "0cb7295bbc7a4c9c94a9486d2ec497e2"
format_stamp: "Formatted at 2026-09-26 07:25:15 on dist-test-slave-g04g"
server_key: "f95d491e7e3e597233c4afd90963cdfb"
server_key_iv: "aed51bc121201bb177d39e56d6ae3c05"
server_key_version: "encryptionkey@0"
I20260926 07:25:15.110087 3958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:15.127543 3958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:15.130628 3958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:15.130785 3958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:15.137478 3958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33549
I20260926 07:25:15.137504 4025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33549 every 8 connection(s)
I20260926 07:25:15.138389 3958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:15.138960 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 3958
I20260926 07:25:15.139115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:15.139433 392 external_mini_cluster.cc:1428] Setting key d37763345414735819ee85f32349e7d1
I20260926 07:25:15.141543 4026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:15.146838 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2: Bootstrap starting.
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[3945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:15.149216 4026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:15.149935 4026 log.cc:824] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:15.151821 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2: No bootstrap required, opened a new log
I20260926 07:25:15.152838 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:15.140769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46494 (local address 127.0.98.62:33549)
0926 07:25:15.141191 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:15.141206 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:15.141241 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:25:15.141913 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:15.141929 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:15.142266 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:15.142588 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:15.143358 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.144309 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.145127 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.145585 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.145776 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:15.148252 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:15.148257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:15.148269 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:15.148305 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:15.149966 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:15.150527 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:15.150537 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:15.150541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:15.150648 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:15.150932 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:15.150936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:15.150938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:15.151376 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:15.151569 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:15.151828 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 07:25:15.152143 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":135,"threads_started":1}
I20260926 07:25:15.155694 4026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb7295bbc7a4c9c94a9486d2ec497e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } }
I20260926 07:25:15.155956 4026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:15.156020 4026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb7295bbc7a4c9c94a9486d2ec497e2, State: Initialized, Role: FOLLOWER
I20260926 07:25:15.156435 4026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb7295bbc7a4c9c94a9486d2ec497e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } }
I20260926 07:25:15.156574 4026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:15.156651 4026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:15.156745 4026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:15.157641 4026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb7295bbc7a4c9c94a9486d2ec497e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } }
I20260926 07:25:15.157999 4026 leader_election.cc:302] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cb7295bbc7a4c9c94a9486d2ec497e2; no voters:
I20260926 07:25:15.158330 4026 leader_election.cc:288] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:15.158461 4031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:15.158694 4031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [term 1 LEADER]: Becoming Leader. State: Replica: 0cb7295bbc7a4c9c94a9486d2ec497e2, State: Running, Role: LEADER
I20260926 07:25:15.159065 4031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb7295bbc7a4c9c94a9486d2ec497e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } }
I20260926 07:25:15.159535 4026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:15.160672 4033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cb7295bbc7a4c9c94a9486d2ec497e2. Latest consensus state: current_term: 1 leader_uuid: "0cb7295bbc7a4c9c94a9486d2ec497e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb7295bbc7a4c9c94a9486d2ec497e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } } }
I20260926 07:25:15.160804 4033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:15.161116 4032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cb7295bbc7a4c9c94a9486d2ec497e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb7295bbc7a4c9c94a9486d2ec497e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33549 } } }
I20260926 07:25:15.161164 4040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:15.161268 4032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:15.164053 4040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:15.169013 4040 catalog_manager.cc:1347] Generated new cluster ID: 51fcc68064a44448bd176eb95f5ac9b0
I20260926 07:25:15.169080 4040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:15.175904 4040 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:15.176739 4040 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:15.189385 4040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cb7295bbc7a4c9c94a9486d2ec497e2: Generated new TSK 0
I20260926 07:25:15.190155 4040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:25:15.223450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:15.334288 4052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:15.334540 4052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:15.334584 4052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:15.338522 4052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:15.338603 4052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:15.338663 4052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:15.343674 4052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:15.344596 4052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:15.345521 4052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:15.354843 4057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:15.354807 4058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:15.354854 4052 server_base.cc:1034] running on GCE node
W20260926 07:25:15.354803 4060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:15.355443 4052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:15.356029 4052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:15.357182 4052 hybrid_clock.cc:648] HybridClock initialized: now 1790407515357169 us; error 27 us; skew 500 ppm
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:15.359992 4052 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:25:15.360773 4052 webserver.cc:460] Webserver started at http://127.0.98.1:35965/ using document root <none> and password file <none>
I20260926 07:25:15.361363 4052 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:15.361423 4052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:15.361632 4052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:15.363691 4052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "73d786e5b24f4b2ab492d743abc58a90"
format_stamp: "Formatted at 2026-09-26 07:25:15 on dist-test-slave-g04g"
server_key: "6cd5631c51bca8c2cec66ab3785e3197"
server_key_iv: "d2fe38b21ca4c0a07d869fa08343f676"
server_key_version: "encryptionkey@0"
I20260926 07:25:15.364174 4052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "73d786e5b24f4b2ab492d743abc58a90"
format_stamp: "Formatted at 2026-09-26 07:25:15 on dist-test-slave-g04g"
server_key: "6cd5631c51bca8c2cec66ab3785e3197"
server_key_iv: "d2fe38b21ca4c0a07d869fa08343f676"
server_key_version: "encryptionkey@0"
I20260926 07:25:15.367565 4052 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:15.369794 4067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:15.370944 4052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:15.371085 4052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "73d786e5b24f4b2ab492d743abc58a90"
format_stamp: "Formatted at 2026-09-26 07:25:15 on dist-test-slave-g04g"
server_key: "6cd5631c51bca8c2cec66ab3785e3197"
server_key_iv: "d2fe38b21ca4c0a07d869fa08343f676"
server_key_version: "encryptionkey@0"
I20260926 07:25:15.371182 4052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:15.404214 4052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:15.407722 4052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:15.407929 4052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:15.408540 4052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:15.409381 4052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:15.409426 4052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:15.409490 4052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:15.409548 4052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:15.418437 4052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40663
I20260926 07:25:15.418458 4180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40663 every 8 connection(s)
I20260926 07:25:15.419368 4052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:15.419898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 4052
I20260926 07:25:15.420022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:25:15.420337 392 external_mini_cluster.cc:1428] Setting key 46ff49367b9682e8e4ec409952741bbd
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[3945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:15.437716 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:15.421207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48653 (local address 127.0.98.62:33549)
0926 07:25:15.421334 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:15.421339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:15.422399 (+ 1060us) server_negotiation.cc:407] Connection header received
0926 07:25:15.424156 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:15.424161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:15.424221 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:15.424322 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:15.425616 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.426521 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.427798 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.428184 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.428320 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:15.433685 (+ 5365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:15.433688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:15.433690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:15.433715 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:15.435241 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:15.435949 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:15.435952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:15.435953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:15.436034 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:15.436470 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:15.436472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:15.436473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:15.436661 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:15.436785 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:15.437471 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 07:25:15.437583 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:25:15.439101 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:15.421710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33549 (local address 127.0.98.1:48653)
0926 07:25:15.422226 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:15.422289 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:15.423934 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:15.424515 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:15.424523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:15.424925 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:15.425461 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:15.425476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.426676 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.426679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:15.427673 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:15.427683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.428369 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.428372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:15.428576 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:15.429335 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:15.429354 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:15.433451 (+ 4097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:15.435376 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:15.435385 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:15.435407 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:15.435805 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:15.436152 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:15.436158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:15.436163 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:15.436358 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:15.436802 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:15.436812 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:15.437194 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 07:25:15.437416 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 07:25:15.437829 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260926 07:25:15.440639 4181 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33549
I20260926 07:25:15.440912 4181 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:15.441437 4181 heartbeater.cc:511] Master 127.0.98.62:33549 requested a full tablet report, sending...
I20260926 07:25:15.443068 3990 ts_manager.cc:194] Registered new tserver with Master: 73d786e5b24f4b2ab492d743abc58a90 (127.0.98.1:40663)
I20260926 07:25:15.444355 3990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:25:15.450350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:15.558149 4186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:15.558439 4186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:15.558511 4186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:15.562769 4186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:15.562855 4186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:15.562942 4186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:15.568221 4186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:15.569319 4186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:15.570344 4186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:15.581429 4194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:15.581498 4192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:15.581629 4186 server_base.cc:1034] running on GCE node
W20260926 07:25:15.581449 4191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:15.582082 4186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:15.582703 4186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:15.583863 4186 hybrid_clock.cc:648] HybridClock initialized: now 1790407515583848 us; error 26 us; skew 500 ppm
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:15.586515 4186 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:25:15.587427 4186 webserver.cc:460] Webserver started at http://127.0.98.2:34923/ using document root <none> and password file <none>
I20260926 07:25:15.588022 4186 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:15.588085 4186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:15.588315 4186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:15.590265 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "f7679b5823fc442ebfea8b57380810d4"
format_stamp: "Formatted at 2026-09-26 07:25:15 on dist-test-slave-g04g"
server_key: "00f1906b2bd7801c413bd663072a716a"
server_key_iv: "3f489db3d21d50117ce81a59a2de86d6"
server_key_version: "encryptionkey@0"
I20260926 07:25:15.590797 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f7679b5823fc442ebfea8b57380810d4"
format_stamp: "Formatted at 2026-09-26 07:25:15 on dist-test-slave-g04g"
server_key: "00f1906b2bd7801c413bd663072a716a"
server_key_iv: "3f489db3d21d50117ce81a59a2de86d6"
server_key_version: "encryptionkey@0"
I20260926 07:25:15.594290 4186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:25:15.596570 4201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:15.597728 4186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 07:25:15.597848 4186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "f7679b5823fc442ebfea8b57380810d4"
format_stamp: "Formatted at 2026-09-26 07:25:15 on dist-test-slave-g04g"
server_key: "00f1906b2bd7801c413bd663072a716a"
server_key_iv: "3f489db3d21d50117ce81a59a2de86d6"
server_key_version: "encryptionkey@0"
I20260926 07:25:15.597947 4186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:15.616155 4186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:15.619206 4186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:15.619403 4186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:15.620040 4186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:15.620834 4186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:15.620900 4186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:15.620970 4186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:15.621009 4186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:15.629910 4186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45541
I20260926 07:25:15.629932 4314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45541 every 8 connection(s)
I20260926 07:25:15.630795 4186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:25:15.638718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 4186
I20260926 07:25:15.638856 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:25:15.639199 392 external_mini_cluster.cc:1428] Setting key 2adbba4101fdaa366b11fc492d005b40
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[3945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:15.644490 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:15.632554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34983 (local address 127.0.98.62:33549)
0926 07:25:15.632683 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:15.632688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:15.633544 (+ 856us) server_negotiation.cc:407] Connection header received
0926 07:25:15.634894 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:15.634898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:15.634957 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:15.635049 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:15.636201 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.636722 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.637829 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.638062 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.638184 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:15.640826 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:15.640830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:15.640833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:15.640857 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:15.642318 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:15.642801 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:15.642807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:15.642810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:15.642902 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:15.643285 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:15.643291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:15.643295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:15.643533 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:15.643638 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:15.644098 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 07:25:15.644243 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:25:15.645107 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:15.632941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33549 (local address 127.0.98.2:34983)
0926 07:25:15.633393 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:15.633434 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:15.634687 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:15.635219 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:15.635226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:15.635565 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:15.636032 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:15.636044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.636841 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.636845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:15.637707 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:15.637715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.638221 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.638225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:15.638388 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:15.639018 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:15.639081 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:15.640673 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:15.642429 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:15.642434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:15.642444 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:15.642706 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:15.642988 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:15.642990 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:15.642991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:15.643177 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:15.643631 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:15.643636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:15.643857 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 07:25:15.644004 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 07:25:15.644253 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":133,"threads_started":1}
I20260926 07:25:15.646142 4315 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33549
I20260926 07:25:15.646364 4315 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:15.646821 4315 heartbeater.cc:511] Master 127.0.98.62:33549 requested a full tablet report, sending...
I20260926 07:25:15.647807 3990 ts_manager.cc:194] Registered new tserver with Master: f7679b5823fc442ebfea8b57380810d4 (127.0.98.2:45541)
I20260926 07:25:15.648366 3990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34983
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:25:15.666803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:15.775525 4320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:15.775825 4320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:15.775900 4320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:15.780184 4320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:15.780256 4320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:15.780339 4320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:15.785575 4320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:15.786607 4320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:15.787658 4320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:15.797299 4325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:15.797299 4328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:15.797413 4320 server_base.cc:1034] running on GCE node
W20260926 07:25:15.797370 4326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:15.798053 4320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:15.798733 4320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:15.799901 4320 hybrid_clock.cc:648] HybridClock initialized: now 1790407515799880 us; error 29 us; skew 500 ppm
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:15.802865 4320 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:25:15.803946 4320 webserver.cc:460] Webserver started at http://127.0.98.3:38795/ using document root <none> and password file <none>
I20260926 07:25:15.804600 4320 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:15.804678 4320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:15.804854 4320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:15.807020 4320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "5758c0478d0f428d9c1d79456050acea"
format_stamp: "Formatted at 2026-09-26 07:25:15 on dist-test-slave-g04g"
server_key: "493ddc11db2a6c745171ece28661aafe"
server_key_iv: "100f3706446474205554d88f88e336a2"
server_key_version: "encryptionkey@0"
I20260926 07:25:15.807682 4320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5758c0478d0f428d9c1d79456050acea"
format_stamp: "Formatted at 2026-09-26 07:25:15 on dist-test-slave-g04g"
server_key: "493ddc11db2a6c745171ece28661aafe"
server_key_iv: "100f3706446474205554d88f88e336a2"
server_key_version: "encryptionkey@0"
I20260926 07:25:15.811506 4320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260926 07:25:15.814190 4335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:15.815508 4320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:15.815673 4320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "5758c0478d0f428d9c1d79456050acea"
format_stamp: "Formatted at 2026-09-26 07:25:15 on dist-test-slave-g04g"
server_key: "493ddc11db2a6c745171ece28661aafe"
server_key_iv: "100f3706446474205554d88f88e336a2"
server_key_version: "encryptionkey@0"
I20260926 07:25:15.815788 4320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:15.832422 4320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:15.835546 4320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:15.835755 4320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:15.836340 4320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:15.837172 4320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:15.837244 4320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:15.837316 4320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:15.837360 4320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:15.846719 4320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39297
I20260926 07:25:15.846753 4448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39297 every 8 connection(s)
I20260926 07:25:15.847688 4320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:15.852469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 4320
I20260926 07:25:15.852568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:15.852856 392 external_mini_cluster.cc:1428] Setting key 6317f63bf100465e7b5bc6c8ac4b80d4
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[3945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:15.861441 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:15.849483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57137 (local address 127.0.98.62:33549)
0926 07:25:15.849639 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:15.849644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:15.850529 (+ 885us) server_negotiation.cc:407] Connection header received
0926 07:25:15.851954 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:15.851957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:15.852005 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:15.852085 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:15.853365 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.853970 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.855214 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.855460 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.855568 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:15.857901 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:15.857905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:15.857907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:15.857934 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:15.859406 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:15.859886 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:15.859891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:15.859893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:15.860015 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:15.860380 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:15.860383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:15.860385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:15.860556 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:15.860673 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:15.861154 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 07:25:15.861277 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:25:15.862176 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:15.849909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33549 (local address 127.0.98.3:57137)
0926 07:25:15.850377 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:15.850419 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:15.851761 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:15.852232 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:15.852239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:15.852726 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:15.853222 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:15.853235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.854087 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.854092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:15.855099 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:15.855108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.855607 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.855611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:15.855766 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:15.856359 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:15.856375 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:15.857747 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:15.859516 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:15.859521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:15.859536 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:15.859786 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:15.860133 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:15.860136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:15.860139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:15.860282 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:15.860669 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:15.860675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:15.860903 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 07:25:15.861104 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 07:25:15.861403 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":115,"threads_started":1}
I20260926 07:25:15.863229 4449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33549
I20260926 07:25:15.863432 4449 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:15.863888 4449 heartbeater.cc:511] Master 127.0.98.62:33549 requested a full tablet report, sending...
I20260926 07:25:15.864773 3990 ts_manager.cc:194] Registered new tserver with Master: 5758c0478d0f428d9c1d79456050acea (127.0.98.3:39297)
I20260926 07:25:15.865368 3990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57137
I20260926 07:25:15.866477 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[3945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[3945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 07:25:15.892867 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:15.883575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.0.98.1:40663)
0926 07:25:15.883840 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:15.883857 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:15.883885 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:25:15.884047 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:15.884053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:15.884261 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:15.884412 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:15.885069 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.885846 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.886672 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.887092 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.887202 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:15.889601 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:15.889608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:15.889627 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:15.889650 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:15.891174 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:15.891575 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:15.891586 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:15.891588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:15.891669 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:15.891924 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:15.891929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:15.891931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:15.892166 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:15.892393 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:15.892540 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 07:25:15.892699 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":64,"threads_started":1}
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[3945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:15.904238 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:15.895560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.0.98.62:33549)
0926 07:25:15.895737 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:15.895742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:15.895756 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:15.895886 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:15.895889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:15.895935 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:15.896014 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:15.896576 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.897083 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.897832 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:15.898048 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:15.898152 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:15.900766 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:15.900770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:15.900772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:15.900817 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:15.902372 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:15.902819 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:15.902828 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:15.902830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:15.902909 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:15.903250 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:15.903255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:15.903257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:15.903490 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:15.903640 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:15.903897 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 07:25:15.904028 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260926 07:25:15.904687 3990 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46510
I20260926 07:25:15.905289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 4052
I20260926 07:25:15.911654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 4186
I20260926 07:25:15.917182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 4320
I20260926 07:25:15.922905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 3958
2026-09-26T07:25:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2117 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[4461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[4461](info): set up 2 sockets
Sep 26 07:25:15 dist-test-slave-g04g krb5kdc[4461](info): commencing operation
krb5kdc: starting...
W20260926 07:25:16.959177 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:25:17 dist-test-slave-g04g krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:25:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:25:17.078557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37939
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:17.185133 4474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:17.185391 4474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:17.185434 4474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:17.189535 4474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:17.189601 4474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:17.189621 4474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:17.189638 4474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:17.189653 4474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:17.194846 4474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:17.195900 4474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:17.196870 4474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:17.204823 4480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:17.204982 4474 server_base.cc:1034] running on GCE node
W20260926 07:25:17.204854 4482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:17.204849 4479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:17.205644 4474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:17.206700 4474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:17.207854 4474 hybrid_clock.cc:648] HybridClock initialized: now 1790407517207814 us; error 53 us; skew 500 ppm
Sep 26 07:25:17 dist-test-slave-g04g krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:17.215317 4474 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:25:17.216073 4474 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:17.216144 4474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:17.216379 4474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:17.218413 4474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "cdc5aad5674540f198cdda330448f614"
format_stamp: "Formatted at 2026-09-26 07:25:17 on dist-test-slave-g04g"
server_key: "c877accf49f54298f578c72974f9c905"
server_key_iv: "f0eaa5a32eaf9af88f006bfeeb731c37"
server_key_version: "encryptionkey@0"
I20260926 07:25:17.218961 4474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "cdc5aad5674540f198cdda330448f614"
format_stamp: "Formatted at 2026-09-26 07:25:17 on dist-test-slave-g04g"
server_key: "c877accf49f54298f578c72974f9c905"
server_key_iv: "f0eaa5a32eaf9af88f006bfeeb731c37"
server_key_version: "encryptionkey@0"
I20260926 07:25:17.222507 4474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:25:17.224798 4488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:17.225950 4474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:17.226105 4474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "cdc5aad5674540f198cdda330448f614"
format_stamp: "Formatted at 2026-09-26 07:25:17 on dist-test-slave-g04g"
server_key: "c877accf49f54298f578c72974f9c905"
server_key_iv: "f0eaa5a32eaf9af88f006bfeeb731c37"
server_key_version: "encryptionkey@0"
I20260926 07:25:17.226193 4474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:17.255316 4474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:17.258322 4474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:17.258533 4474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:17.265509 4474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37939
I20260926 07:25:17.265790 4540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37939 every 8 connection(s)
I20260926 07:25:17.266175 4474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:17.269136 4541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:17.274184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 4474
I20260926 07:25:17.274328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:17.274634 392 external_mini_cluster.cc:1428] Setting key e25d86e563df68b2df52ed035ed3e32f
I20260926 07:25:17.275323 4541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614: Bootstrap starting.
I20260926 07:25:17.277849 4541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:17.278844 4541 log.cc:824] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:17.281040 4541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614: No bootstrap required, opened a new log
Sep 26 07:25:17 dist-test-slave-g04g krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:17.285660 4541 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5aad5674540f198cdda330448f614" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } }
I20260926 07:25:17.285931 4541 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:17.286018 4541 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc5aad5674540f198cdda330448f614, State: Initialized, Role: FOLLOWER
I20260926 07:25:17.286480 4541 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5aad5674540f198cdda330448f614" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } }
I20260926 07:25:17.286644 4541 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:17.286733 4541 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:17.286857 4541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:17.288040 4541 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5aad5674540f198cdda330448f614" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } }
I20260926 07:25:17.288475 4541 leader_election.cc:302] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc5aad5674540f198cdda330448f614; no voters:
I20260926 07:25:17.288859 4541 leader_election.cc:288] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:17.288967 4546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:17.289188 4546 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [term 1 LEADER]: Becoming Leader. State: Replica: cdc5aad5674540f198cdda330448f614, State: Running, Role: LEADER
I20260926 07:25:17.289564 4546 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5aad5674540f198cdda330448f614" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } }
I20260926 07:25:17.290043 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:17.276030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.0.98.62:37939)
0926 07:25:17.276431 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:17.276448 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:17.276482 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:25:17.277095 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:17.277112 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:17.277437 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:17.277719 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:17.278669 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.279927 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.281160 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.281642 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.281758 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:17.284707 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:17.284712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:17.284725 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:17.284752 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:17.286672 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:17.287245 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:17.287250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:17.287254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:17.287365 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:17.287653 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:17.287655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:17.287656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:17.288039 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:17.288224 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:17.288553 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 07:25:17.289048 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":132,"threads_started":1}
I20260926 07:25:17.290249 4541 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:17.292223 4548 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc5aad5674540f198cdda330448f614. Latest consensus state: current_term: 1 leader_uuid: "cdc5aad5674540f198cdda330448f614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5aad5674540f198cdda330448f614" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } }
I20260926 07:25:17.292397 4547 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdc5aad5674540f198cdda330448f614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5aad5674540f198cdda330448f614" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } }
I20260926 07:25:17.292457 4548 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:17.292538 4547 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614 [sys.catalog]: This master's current role is: LEADER
W20260926 07:25:17.295673 4560 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:25:17.295761 4560 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:25:17.295826 4561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:17.296711 4561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:17.302508 4561 catalog_manager.cc:1347] Generated new cluster ID: 31a56805421d4762b6a1b91dc2ab53e3
I20260926 07:25:17.302579 4561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:17.311748 4561 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:17.312634 4561 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:17.316625 4561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614: Generated new TSK 0
I20260926 07:25:17.317231 4561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:25:17.350415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37939
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:17.459722 4567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:17.459972 4567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:17.460018 4567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:17.464075 4567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:17.464133 4567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:17.464191 4567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:17.469339 4567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:17.470402 4567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:17.471498 4567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:17.480897 4572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:17.481236 4575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:17.481236 4573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:17.481345 4567 server_base.cc:1034] running on GCE node
I20260926 07:25:17.481721 4567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:17.482339 4567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:17.483515 4567 hybrid_clock.cc:648] HybridClock initialized: now 1790407517483497 us; error 31 us; skew 500 ppm
Sep 26 07:25:17 dist-test-slave-g04g krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:17.486176 4567 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:25:17.486907 4567 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:17.486994 4567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:17.487246 4567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:17.489236 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "3ca03eed0b804bc584048fa1b2b72f8e"
format_stamp: "Formatted at 2026-09-26 07:25:17 on dist-test-slave-g04g"
server_key: "5072f21d0914ebae756e20468c2cf78a"
server_key_iv: "21987c5165ca82231a1deb8c64683bf0"
server_key_version: "encryptionkey@0"
I20260926 07:25:17.489781 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3ca03eed0b804bc584048fa1b2b72f8e"
format_stamp: "Formatted at 2026-09-26 07:25:17 on dist-test-slave-g04g"
server_key: "5072f21d0914ebae756e20468c2cf78a"
server_key_iv: "21987c5165ca82231a1deb8c64683bf0"
server_key_version: "encryptionkey@0"
I20260926 07:25:17.493481 4567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 07:25:17.495777 4581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:17.496894 4567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 07:25:17.497014 4567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "3ca03eed0b804bc584048fa1b2b72f8e"
format_stamp: "Formatted at 2026-09-26 07:25:17 on dist-test-slave-g04g"
server_key: "5072f21d0914ebae756e20468c2cf78a"
server_key_iv: "21987c5165ca82231a1deb8c64683bf0"
server_key_version: "encryptionkey@0"
I20260926 07:25:17.497118 4567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:17.520867 4567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:17.523716 4567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:17.523902 4567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:17.524432 4567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:17.525208 4567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:17.525276 4567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:17.525341 4567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:17.525388 4567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:17.534333 4567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42565
I20260926 07:25:17.534356 4694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42565 every 8 connection(s)
I20260926 07:25:17.534916 4567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:17.536388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 4567
I20260926 07:25:17.536513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:25:17.536840 392 external_mini_cluster.cc:1428] Setting key 7a58d837233ec1845f440a6ca606dda0
Sep 26 07:25:17 dist-test-slave-g04g krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 07:25:17.550063 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:17.536548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39217 (local address 127.0.98.62:37939)
0926 07:25:17.536697 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:17.536703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:17.537590 (+ 887us) server_negotiation.cc:407] Connection header received
0926 07:25:17.539462 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:17.539464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:17.539511 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:17.539584 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:17.540900 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.541416 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.543275 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.543574 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.543682 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:17.546509 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:17.546512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:17.546514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:17.546536 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:17.548026 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:17.548572 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:17.548579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:17.548582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:17.548652 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:17.548988 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:17.548991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:17.548993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:17.549174 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:17.549274 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:17.549768 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 07:25:17.549891 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 07:25:17.551182 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:17.536932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.1:39217)
0926 07:25:17.537440 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:17.537496 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:17.539242 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:17.539766 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:17.539778 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:17.540282 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:17.540759 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:17.540772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.541544 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.541550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:17.543150 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:17.543165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.543726 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.543733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:17.543969 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:17.544663 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:17.544690 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:17.546342 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:17.548180 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:17.548186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:17.548201 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:17.548456 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:17.548747 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:17.548750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:17.548752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:17.548889 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:17.549282 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:17.549291 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:17.549535 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 07:25:17.549717 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 07:25:17.550036 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":112,"threads_started":1}
I20260926 07:25:17.552461 4695 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37939
I20260926 07:25:17.552750 4695 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:17.553280 4695 heartbeater.cc:511] Master 127.0.98.62:37939 requested a full tablet report, sending...
I20260926 07:25:17.554961 4505 ts_manager.cc:194] Registered new tserver with Master: 3ca03eed0b804bc584048fa1b2b72f8e (127.0.98.1:42565)
I20260926 07:25:17.556695 4505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:25:17.566592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37939
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:17.676060 4700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:17.676312 4700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:17.676357 4700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:17.681154 4700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:17.681213 4700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:17.681272 4700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:17.686295 4700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:17.687304 4700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:17.688271 4700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:17.697744 4706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:17.697744 4708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:17.697747 4705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:17.698099 4700 server_base.cc:1034] running on GCE node
I20260926 07:25:17.698464 4700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:17.699116 4700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:17.700287 4700 hybrid_clock.cc:648] HybridClock initialized: now 1790407517700258 us; error 47 us; skew 500 ppm
Sep 26 07:25:17 dist-test-slave-g04g krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:17.703164 4700 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:25:17.703879 4700 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:17.703969 4700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:17.704173 4700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:17.706183 4700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "b4c1883c031c425ab8ab6addf8d427a2"
format_stamp: "Formatted at 2026-09-26 07:25:17 on dist-test-slave-g04g"
server_key: "5b171bbff12d122c9711cdf0ceb6f046"
server_key_iv: "049ec2ccc08e6a6aa3b9d9fb227b1fef"
server_key_version: "encryptionkey@0"
I20260926 07:25:17.706804 4700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b4c1883c031c425ab8ab6addf8d427a2"
format_stamp: "Formatted at 2026-09-26 07:25:17 on dist-test-slave-g04g"
server_key: "5b171bbff12d122c9711cdf0ceb6f046"
server_key_iv: "049ec2ccc08e6a6aa3b9d9fb227b1fef"
server_key_version: "encryptionkey@0"
I20260926 07:25:17.710618 4700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:25:17.713032 4714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:17.714458 4700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:17.714701 4700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "b4c1883c031c425ab8ab6addf8d427a2"
format_stamp: "Formatted at 2026-09-26 07:25:17 on dist-test-slave-g04g"
server_key: "5b171bbff12d122c9711cdf0ceb6f046"
server_key_iv: "049ec2ccc08e6a6aa3b9d9fb227b1fef"
server_key_version: "encryptionkey@0"
I20260926 07:25:17.714812 4700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:17.733235 4700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:17.736155 4700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:17.736351 4700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:17.736922 4700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:17.737754 4700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:17.737823 4700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:17.737890 4700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:17.737936 4700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:17.746676 4700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43065
I20260926 07:25:17.746694 4827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43065 every 8 connection(s)
I20260926 07:25:17.747313 4700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:25:17.752746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 4700
I20260926 07:25:17.752884 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:25:17.753162 392 external_mini_cluster.cc:1428] Setting key 713d3195db073806bd3be7dae49cda6c
Sep 26 07:25:17 dist-test-slave-g04g krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:17.761399 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:17.748985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52347 (local address 127.0.98.62:37939)
0926 07:25:17.749159 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:17.749163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:17.749989 (+ 826us) server_negotiation.cc:407] Connection header received
0926 07:25:17.751390 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:17.751394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:17.751500 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:17.751591 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:17.752717 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.753372 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.754669 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.754901 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.755007 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:17.757711 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:17.757718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:17.757724 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:17.757756 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:17.759349 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:17.759836 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:17.759845 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:17.759846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:17.759919 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:17.760265 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:17.760268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:17.760270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:17.760467 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:17.760563 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:17.761186 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 07:25:17.761282 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 07:25:17.762575 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:17.749360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.2:52347)
0926 07:25:17.749820 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:17.749862 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:17.751192 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:17.751754 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:17.751761 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:17.752121 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:17.752580 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:17.752593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.753502 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.753505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:17.754557 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:17.754567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.755063 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.755067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:17.755226 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:17.755839 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:17.755855 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:17.757547 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:17.759447 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:17.759452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:17.759463 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:17.759721 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:17.760017 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:17.760023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:17.760027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:17.760163 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:17.760591 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:17.760601 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:17.760914 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 07:25:17.761132 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 07:25:17.761497 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 07:25:17.763743 4828 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37939
I20260926 07:25:17.763985 4828 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:17.764396 4828 heartbeater.cc:511] Master 127.0.98.62:37939 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 07:25:17.765398 4505 ts_manager.cc:194] Registered new tserver with Master: b4c1883c031c425ab8ab6addf8d427a2 (127.0.98.2:43065)
I20260926 07:25:17.766194 4505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:25:17.782900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37939
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:17.893044 4833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:17.893332 4833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:17.893402 4833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:17.897518 4833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:17.897596 4833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:17.897683 4833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:17.903209 4833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:17.904321 4833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:17.905335 4833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:17.914894 4841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:17.914894 4838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:17.914964 4839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:17.915089 4833 server_base.cc:1034] running on GCE node
I20260926 07:25:17.915578 4833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:17.916186 4833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:17.917338 4833 hybrid_clock.cc:648] HybridClock initialized: now 1790407517917291 us; error 61 us; skew 500 ppm
Sep 26 07:25:17 dist-test-slave-g04g krb5kdc[4461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:17.920166 4833 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:25:17.920885 4833 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:17.920970 4833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:17.921195 4833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:17.923242 4833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "02e236a9af8d48f693f766c482236987"
format_stamp: "Formatted at 2026-09-26 07:25:17 on dist-test-slave-g04g"
server_key: "c155dcbacccba35a421fe46b884c8baa"
server_key_iv: "a4f9066d104e07079c295ceb0866179f"
server_key_version: "encryptionkey@0"
I20260926 07:25:17.923790 4833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "02e236a9af8d48f693f766c482236987"
format_stamp: "Formatted at 2026-09-26 07:25:17 on dist-test-slave-g04g"
server_key: "c155dcbacccba35a421fe46b884c8baa"
server_key_iv: "a4f9066d104e07079c295ceb0866179f"
server_key_version: "encryptionkey@0"
I20260926 07:25:17.927309 4833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:17.929664 4847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:17.930871 4833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:17.930995 4833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "02e236a9af8d48f693f766c482236987"
format_stamp: "Formatted at 2026-09-26 07:25:17 on dist-test-slave-g04g"
server_key: "c155dcbacccba35a421fe46b884c8baa"
server_key_iv: "a4f9066d104e07079c295ceb0866179f"
server_key_version: "encryptionkey@0"
I20260926 07:25:17.931156 4833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:17.947445 4833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:17.950870 4833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:17.951108 4833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:17.951685 4833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:17.952533 4833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:17.952600 4833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:17.952666 4833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:17.952713 4833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:17.961427 4833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42643
I20260926 07:25:17.961452 4960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42643 every 8 connection(s)
I20260926 07:25:17.962023 4833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:17.968591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 4833
I20260926 07:25:17.968710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:17.968997 392 external_mini_cluster.cc:1428] Setting key eb7ff690e6e189706835ce41a266a180
Sep 26 07:25:17 dist-test-slave-g04g krb5kdc[4461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:17.976547 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:17.964045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50737 (local address 127.0.98.62:37939)
0926 07:25:17.964202 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:17.964207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:17.965099 (+ 892us) server_negotiation.cc:407] Connection header received
0926 07:25:17.966433 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:17.966436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:17.966480 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:17.966557 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:17.967941 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.968599 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.970145 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.970490 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.970576 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:17.973239 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:17.973242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:17.973244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:17.973266 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:17.974589 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:17.975118 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:17.975121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:17.975122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:17.975211 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:17.975553 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:17.975555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:17.975556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:17.975724 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:17.975803 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:17.976320 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 07:25:17.976430 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:25:17.977315 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:17.964452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.98.3:50737)
0926 07:25:17.964956 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:17.965000 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:17.966192 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:17.966783 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:17.966791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:17.967262 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:17.967782 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:17.967796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.968742 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.968746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:17.970011 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:17.970021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.970643 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.970648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:17.970877 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:17.971576 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:17.971595 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:17.973093 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:17.974697 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:17.974701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:17.974713 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:17.974972 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:17.975291 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:17.975294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:17.975295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:17.975455 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:17.975843 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:17.975849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:17.976075 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 07:25:17.976235 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 07:25:17.976491 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260926 07:25:17.978394 4961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37939
I20260926 07:25:17.978655 4961 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:17.979130 4961 heartbeater.cc:511] Master 127.0.98.62:37939 requested a full tablet report, sending...
I20260926 07:25:17.980095 4505 ts_manager.cc:194] Registered new tserver with Master: 02e236a9af8d48f693f766c482236987 (127.0.98.3:42643)
I20260926 07:25:17.980705 4505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50737
I20260926 07:25:17.982893 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:25:17.992702 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:17.984679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51418 (local address 127.0.98.62:37939)
0926 07:25:17.984859 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:17.984864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:17.984998 (+ 134us) server_negotiation.cc:407] Connection header received
0926 07:25:17.985143 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:17.985146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:17.985190 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:17.985272 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:17.985921 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.986434 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.987299 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:17.987517 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:17.987620 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:17.988948 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:17.988952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:17.988955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:17.988979 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:17.990669 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:17.991295 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:17.991299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:17.991300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:17.991387 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:17.991639 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:17.991642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:17.991643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:17.991849 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:17.992011 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:17.992311 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 07:25:17.992501 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:25:17.995699 4505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:25:17.998704 4505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:18.013296 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.007175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42643 (local address 127.0.0.1:54820)
0926 07:25:18.007982 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:18.008028 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:18.008182 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:18.008799 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:18.008804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:18.008826 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:18.008955 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:18.008974 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.010448 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.010451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:18.011832 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:18.011841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.012756 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.012762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:18.012887 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:18.012926 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:25:18.013026 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:25:18.013159 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260926 07:25:18.013460 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.008453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.0.98.2:43065)
0926 07:25:18.008796 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:18.008807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:18.009174 (+ 367us) server_negotiation.cc:407] Connection header received
0926 07:25:18.009293 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:18.009298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:18.009482 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:18.009696 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:18.010024 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.011064 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.012090 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.012796 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.012893 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:18.013023 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:18.013191 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 07:25:18.013280 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":158,"threads_started":1}
I20260926 07:25:18.013731 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.007193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42565 (local address 127.0.0.1:51080)
0926 07:25:18.008841 (+ 1648us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:18.008854 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:18.008969 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:18.009382 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:18.009384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:18.009400 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:18.009635 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:18.009647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.011217 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.011220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:18.012327 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:18.012336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.013353 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.013356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:18.013439 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:18.013469 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:25:18.013534 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 07:25:18.013601 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1587,"thread_start_us":90,"threads_started":1}
I20260926 07:25:18.013854 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.007517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43065 (local address 127.0.0.1:58492)
0926 07:25:18.009052 (+ 1535us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:18.009103 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:18.009188 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:18.009836 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:18.009838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:18.009851 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:18.009910 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:18.009914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.011183 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.011190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:18.011969 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:18.011978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.012951 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.012955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:18.013052 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:18.013074 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:25:18.013152 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:25:18.013223 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"thread_start_us":79,"threads_started":1}
I20260926 07:25:18.014133 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.008017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.0.98.1:42565)
0926 07:25:18.008331 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:18.008338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:18.008944 (+ 606us) server_negotiation.cc:407] Connection header received
0926 07:25:18.009072 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:18.009077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:18.009250 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:18.009406 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:18.009730 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.011070 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.012448 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.013217 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.013321 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:18.013791 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:18.013842 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 07:25:18.013946 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":134,"threads_started":1}
I20260926 07:25:18.014281 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.007294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.0.98.3:42643)
0926 07:25:18.007883 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:18.007895 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:18.008323 (+ 428us) server_negotiation.cc:407] Connection header received
0926 07:25:18.008400 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:18.008404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:18.008633 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:18.008805 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:18.009117 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.010292 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.011956 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.012640 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.012744 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:18.012876 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:18.013671 (+ 795us) server_negotiation.cc:299] Negotiation successful
0926 07:25:18.013768 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":147,"threads_started":1}
I20260926 07:25:18.016634 4895 tablet_service.cc:1467] Processing CreateTablet for tablet 8351b05b3b5042cdaa34db3ef2a4a2a4 (DEFAULT_TABLE table=test-table [id=ba0fed631baf4d9c836f0d436ccce1f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:18.016783 4629 tablet_service.cc:1467] Processing CreateTablet for tablet 8351b05b3b5042cdaa34db3ef2a4a2a4 (DEFAULT_TABLE table=test-table [id=ba0fed631baf4d9c836f0d436ccce1f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:18.016784 4762 tablet_service.cc:1467] Processing CreateTablet for tablet 8351b05b3b5042cdaa34db3ef2a4a2a4 (DEFAULT_TABLE table=test-table [id=ba0fed631baf4d9c836f0d436ccce1f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:18.017637 4895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8351b05b3b5042cdaa34db3ef2a4a2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:18.017748 4629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8351b05b3b5042cdaa34db3ef2a4a2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:18.017748 4762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8351b05b3b5042cdaa34db3ef2a4a2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:18.023284 4978 tablet_bootstrap.cc:492] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987: Bootstrap starting.
I20260926 07:25:18.024705 4979 tablet_bootstrap.cc:492] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2: Bootstrap starting.
I20260926 07:25:18.025314 4980 tablet_bootstrap.cc:492] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e: Bootstrap starting.
I20260926 07:25:18.025836 4978 tablet_bootstrap.cc:654] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:18.026744 4979 tablet_bootstrap.cc:654] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:18.026955 4980 tablet_bootstrap.cc:654] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:18.027176 4978 log.cc:824] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:18.027876 4979 log.cc:824] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:18.028132 4980 log.cc:824] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:18.029315 4978 tablet_bootstrap.cc:492] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987: No bootstrap required, opened a new log
I20260926 07:25:18.029507 4978 ts_tablet_manager.cc:1397] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260926 07:25:18.029711 4979 tablet_bootstrap.cc:492] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2: No bootstrap required, opened a new log
I20260926 07:25:18.029886 4979 ts_tablet_manager.cc:1397] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:25:18.032619 4980 tablet_bootstrap.cc:492] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e: No bootstrap required, opened a new log
I20260926 07:25:18.032792 4980 ts_tablet_manager.cc:1397] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260926 07:25:18.032814 4979 raft_consensus.cc:348] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e236a9af8d48f693f766c482236987" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42643 } } peers { permanent_uuid: "3ca03eed0b804bc584048fa1b2b72f8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42565 } } peers { permanent_uuid: "b4c1883c031c425ab8ab6addf8d427a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43065 } }
I20260926 07:25:18.032823 4978 raft_consensus.cc:348] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e236a9af8d48f693f766c482236987" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42643 } } peers { permanent_uuid: "3ca03eed0b804bc584048fa1b2b72f8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42565 } } peers { permanent_uuid: "b4c1883c031c425ab8ab6addf8d427a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43065 } }
I20260926 07:25:18.033075 4979 raft_consensus.cc:374] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:18.033082 4978 raft_consensus.cc:374] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:18.033130 4978 raft_consensus.cc:729] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e236a9af8d48f693f766c482236987, State: Initialized, Role: FOLLOWER
I20260926 07:25:18.033146 4979 raft_consensus.cc:729] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c1883c031c425ab8ab6addf8d427a2, State: Initialized, Role: FOLLOWER
I20260926 07:25:18.033550 4979 consensus_queue.cc:260] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e236a9af8d48f693f766c482236987" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42643 } } peers { permanent_uuid: "3ca03eed0b804bc584048fa1b2b72f8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42565 } } peers { permanent_uuid: "b4c1883c031c425ab8ab6addf8d427a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43065 } }
I20260926 07:25:18.033555 4978 consensus_queue.cc:260] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e236a9af8d48f693f766c482236987" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42643 } } peers { permanent_uuid: "3ca03eed0b804bc584048fa1b2b72f8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42565 } } peers { permanent_uuid: "b4c1883c031c425ab8ab6addf8d427a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43065 } }
I20260926 07:25:18.034453 4978 ts_tablet_manager.cc:1428] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260926 07:25:18.034533 4979 ts_tablet_manager.cc:1428] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 07:25:18.034885 4828 heartbeater.cc:503] Master 127.0.98.62:37939 was elected leader, sending a full tablet report...
I20260926 07:25:18.036361 4961 heartbeater.cc:503] Master 127.0.98.62:37939 was elected leader, sending a full tablet report...
I20260926 07:25:18.036610 4980 raft_consensus.cc:348] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e236a9af8d48f693f766c482236987" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42643 } } peers { permanent_uuid: "3ca03eed0b804bc584048fa1b2b72f8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42565 } } peers { permanent_uuid: "b4c1883c031c425ab8ab6addf8d427a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43065 } }
I20260926 07:25:18.036787 4980 raft_consensus.cc:374] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:18.036819 4980 raft_consensus.cc:729] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca03eed0b804bc584048fa1b2b72f8e, State: Initialized, Role: FOLLOWER
I20260926 07:25:18.037191 4980 consensus_queue.cc:260] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e236a9af8d48f693f766c482236987" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42643 } } peers { permanent_uuid: "3ca03eed0b804bc584048fa1b2b72f8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42565 } } peers { permanent_uuid: "b4c1883c031c425ab8ab6addf8d427a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43065 } }
I20260926 07:25:18.037810 4695 heartbeater.cc:503] Master 127.0.98.62:37939 was elected leader, sending a full tablet report...
I20260926 07:25:18.038089 4980 ts_tablet_manager.cc:1428] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 07:25:18.126258 4985 raft_consensus.cc:482] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:18.126464 4985 raft_consensus.cc:504] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e236a9af8d48f693f766c482236987" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42643 } } peers { permanent_uuid: "3ca03eed0b804bc584048fa1b2b72f8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42565 } } peers { permanent_uuid: "b4c1883c031c425ab8ab6addf8d427a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43065 } }
I20260926 07:25:18.127678 4985 leader_election.cc:288] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ca03eed0b804bc584048fa1b2b72f8e (127.0.98.1:42565), b4c1883c031c425ab8ab6addf8d427a2 (127.0.98.2:43065)
I20260926 07:25:18.132319 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.128050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46051 (local address 127.0.98.2:43065)
0926 07:25:18.128207 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:18.128212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:18.128255 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:25:18.128434 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:18.128437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:18.128492 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:18.128617 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:18.128875 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.129732 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.131230 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.131762 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.131840 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:18.131890 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:18.132107 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 07:25:18.132188 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 07:25:18.132339 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.127947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43065 (local address 127.0.98.3:46051)
0926 07:25:18.128183 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:18.128198 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:18.128314 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:18.128595 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:18.128597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:18.128611 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:18.128770 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:18.128776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.129867 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.129871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:18.131111 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:18.131121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.131877 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.131881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:18.131986 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:18.132014 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:25:18.132093 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:25:18.132182 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":3}
I20260926 07:25:18.132644 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.127947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42565 (local address 127.0.98.3:51113)
0926 07:25:18.128568 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:18.128583 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:18.128686 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:18.128942 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:18.128945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:18.128962 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:18.129089 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:18.129095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.130271 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.130276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:18.131443 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:18.131453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.132204 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.132208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:18.132302 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:18.132325 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:25:18.132417 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:25:18.132488 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":242,"thread_start_us":130,"threads_started":1}
I20260926 07:25:18.133090 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.128109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51113 (local address 127.0.98.1:42565)
0926 07:25:18.128232 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:18.128237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:18.128673 (+ 436us) server_negotiation.cc:407] Connection header received
0926 07:25:18.128786 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:18.128789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:18.128849 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:18.128932 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:18.129185 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.130161 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.131562 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.132092 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.132191 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:18.132245 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:18.132918 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 07:25:18.132981 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:25:18.133040 4782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8351b05b3b5042cdaa34db3ef2a4a2a4" candidate_uuid: "02e236a9af8d48f693f766c482236987" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c1883c031c425ab8ab6addf8d427a2" is_pre_election: true
I20260926 07:25:18.133353 4782 raft_consensus.cc:2454] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e236a9af8d48f693f766c482236987 in term 0.
I20260926 07:25:18.133606 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8351b05b3b5042cdaa34db3ef2a4a2a4" candidate_uuid: "02e236a9af8d48f693f766c482236987" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ca03eed0b804bc584048fa1b2b72f8e" is_pre_election: true
I20260926 07:25:18.133844 4849 leader_election.cc:302] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e236a9af8d48f693f766c482236987, b4c1883c031c425ab8ab6addf8d427a2; no voters:
I20260926 07:25:18.133906 4649 raft_consensus.cc:2454] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e236a9af8d48f693f766c482236987 in term 0.
I20260926 07:25:18.134126 4985 raft_consensus.cc:2781] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:18.134223 4985 raft_consensus.cc:482] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:18.134281 4985 raft_consensus.cc:3037] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:18.135419 4985 raft_consensus.cc:504] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e236a9af8d48f693f766c482236987" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42643 } } peers { permanent_uuid: "3ca03eed0b804bc584048fa1b2b72f8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42565 } } peers { permanent_uuid: "b4c1883c031c425ab8ab6addf8d427a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43065 } }
I20260926 07:25:18.135751 4985 leader_election.cc:288] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [CANDIDATE]: Term 1 election: Requested vote from peers 3ca03eed0b804bc584048fa1b2b72f8e (127.0.98.1:42565), b4c1883c031c425ab8ab6addf8d427a2 (127.0.98.2:43065)
I20260926 07:25:18.136113 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8351b05b3b5042cdaa34db3ef2a4a2a4" candidate_uuid: "02e236a9af8d48f693f766c482236987" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ca03eed0b804bc584048fa1b2b72f8e"
I20260926 07:25:18.136235 4649 raft_consensus.cc:3037] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:18.137218 4649 raft_consensus.cc:2454] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e236a9af8d48f693f766c482236987 in term 1.
I20260926 07:25:18.137534 4782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8351b05b3b5042cdaa34db3ef2a4a2a4" candidate_uuid: "02e236a9af8d48f693f766c482236987" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c1883c031c425ab8ab6addf8d427a2"
I20260926 07:25:18.137568 4851 leader_election.cc:302] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e236a9af8d48f693f766c482236987, 3ca03eed0b804bc584048fa1b2b72f8e; no voters:
I20260926 07:25:18.137670 4782 raft_consensus.cc:3037] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:18.137779 4985 raft_consensus.cc:2781] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:18.138041 4985 raft_consensus.cc:686] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [term 1 LEADER]: Becoming Leader. State: Replica: 02e236a9af8d48f693f766c482236987, State: Running, Role: LEADER
I20260926 07:25:18.138372 4985 consensus_queue.cc:237] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e236a9af8d48f693f766c482236987" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42643 } } peers { permanent_uuid: "3ca03eed0b804bc584048fa1b2b72f8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42565 } } peers { permanent_uuid: "b4c1883c031c425ab8ab6addf8d427a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43065 } }
I20260926 07:25:18.138762 4782 raft_consensus.cc:2454] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e236a9af8d48f693f766c482236987 in term 1.
I20260926 07:25:18.141208 4503 catalog_manager.cc:5495] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02e236a9af8d48f693f766c482236987 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "02e236a9af8d48f693f766c482236987" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e236a9af8d48f693f766c482236987" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ca03eed0b804bc584048fa1b2b72f8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4c1883c031c425ab8ab6addf8d427a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43065 } health_report { overall_health: UNKNOWN } } }
I20260926 07:25:18.168062 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.164942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.0.98.3:42643)
0926 07:25:18.165060 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:18.165064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:18.165078 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:18.165147 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:18.165149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:18.165200 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:18.165287 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:18.165502 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.165978 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.166775 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:18.166975 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:18.167100 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:18.167321 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:18.167743 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:18.167831 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:18.167889 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 07:25:18.167934 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:25:18.173765 4782 raft_consensus.cc:1261] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2 [term 1 FOLLOWER]: Refusing update from remote peer 02e236a9af8d48f693f766c482236987: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:18.173765 4649 raft_consensus.cc:1261] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 3ca03eed0b804bc584048fa1b2b72f8e [term 1 FOLLOWER]: Refusing update from remote peer 02e236a9af8d48f693f766c482236987: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:18.174501 4985 consensus_queue.cc:1035] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ca03eed0b804bc584048fa1b2b72f8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:18.174732 4988 consensus_queue.cc:1035] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c1883c031c425ab8ab6addf8d427a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:18.182183 4992 mvcc.cc:205] Tried to move back new op lower bound from 7333509194432737280 to 7333509194297634816. Current Snapshot: MvccSnapshot[applied={T|T < 7333509194432737280}]
I20260926 07:25:18.191847 4505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:18.192034 4505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:18.194706 4505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdc5aad5674540f198cdda330448f614: Sending DeleteTablet for 3 replicas of tablet 8351b05b3b5042cdaa34db3ef2a4a2a4
I20260926 07:25:18.195574 4762 tablet_service.cc:1514] Processing DeleteTablet for tablet 8351b05b3b5042cdaa34db3ef2a4a2a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58492
I20260926 07:25:18.195590 4629 tablet_service.cc:1514] Processing DeleteTablet for tablet 8351b05b3b5042cdaa34db3ef2a4a2a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51080
I20260926 07:25:18.196019 5001 tablet_replica.cc:330] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2: stopping tablet replica
I20260926 07:25:18.196310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 4567
I20260926 07:25:18.196307 4895 tablet_service.cc:1514] Processing DeleteTablet for tablet 8351b05b3b5042cdaa34db3ef2a4a2a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54820
I20260926 07:25:18.196352 5001 raft_consensus.cc:2229] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:18.196672 5001 raft_consensus.cc:2258] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:18.198094 5003 tablet_replica.cc:330] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987: stopping tablet replica
I20260926 07:25:18.198392 5003 raft_consensus.cc:2229] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:25:18.198686 5001 ts_tablet_manager.cc:1905] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:18.198824 5003 raft_consensus.cc:2258] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:18.200189 5003 ts_tablet_manager.cc:1905] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:18.201423 5001 ts_tablet_manager.cc:1918] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:18.201574 5001 log.cc:1196] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/8351b05b3b5042cdaa34db3ef2a4a2a4
I20260926 07:25:18.201980 5001 ts_tablet_manager.cc:1939] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P b4c1883c031c425ab8ab6addf8d427a2: Deleting consensus metadata
I20260926 07:25:18.202797 5003 ts_tablet_manager.cc:1918] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:18.202960 5003 log.cc:1196] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/8351b05b3b5042cdaa34db3ef2a4a2a4
I20260926 07:25:18.203341 5003 ts_tablet_manager.cc:1939] T 8351b05b3b5042cdaa34db3ef2a4a2a4 P 02e236a9af8d48f693f766c482236987: Deleting consensus metadata
I20260926 07:25:18.203513 4490 catalog_manager.cc:4841] TS b4c1883c031c425ab8ab6addf8d427a2 (127.0.98.2:43065): tablet 8351b05b3b5042cdaa34db3ef2a4a2a4 (table test-table [id=ba0fed631baf4d9c836f0d436ccce1f8]) successfully deleted
W20260926 07:25:18.203788 4492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42565 (error 108)
I20260926 07:25:18.204406 4489 catalog_manager.cc:4841] TS 02e236a9af8d48f693f766c482236987 (127.0.98.3:42643): tablet 8351b05b3b5042cdaa34db3ef2a4a2a4 (table test-table [id=ba0fed631baf4d9c836f0d436ccce1f8]) successfully deleted
I20260926 07:25:18.204718 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.204447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42565 (local address 127.0.0.1:51096)
0926 07:25:18.204566 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:18.204632 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 07:25:18.204807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 4700
W20260926 07:25:18.204879 4492 catalog_manager.cc:4584] TS 3ca03eed0b804bc584048fa1b2b72f8e (127.0.98.1:42565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8351b05b3b5042cdaa34db3ef2a4a2a4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42565: connect: Connection refused (error 111)
I20260926 07:25:18.210484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 4833
I20260926 07:25:18.215212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 4474
2026-09-26T07:25:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2292 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T07:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:18Z Disabled control of system clock
I20260926 07:25:18.244226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44347
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:25:18.351089 5008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:18.351389 5008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:18.351428 5008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:18.355386 5008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:18.355445 5008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:18.355460 5008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:18.355477 5008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:18.355492 5008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:18.360277 5008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:18.361328 5008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:18.362663 5008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:18.370952 5016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:18.371102 5014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:18.371131 5008 server_base.cc:1034] running on GCE node
W20260926 07:25:18.370952 5013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:18.371767 5008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:18.372831 5008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:18.374027 5008 hybrid_clock.cc:648] HybridClock initialized: now 1790407518373994 us; error 48 us; skew 500 ppm
I20260926 07:25:18.376068 5008 webserver.cc:460] Webserver started at http://127.0.98.62:42219/ using document root <none> and password file <none>
I20260926 07:25:18.376643 5008 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:18.376704 5008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:18.376917 5008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:18.378999 5008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "fc996d2b7d2f4ae3b8a8bd06bdb9a970"
format_stamp: "Formatted at 2026-09-26 07:25:18 on dist-test-slave-g04g"
server_key: "39849eec2df759b1b54a580ee344a77d"
server_key_iv: "899beb0002d7060cc329cbaeb4006858"
server_key_version: "encryptionkey@0"
I20260926 07:25:18.379599 5008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc996d2b7d2f4ae3b8a8bd06bdb9a970"
format_stamp: "Formatted at 2026-09-26 07:25:18 on dist-test-slave-g04g"
server_key: "39849eec2df759b1b54a580ee344a77d"
server_key_iv: "899beb0002d7060cc329cbaeb4006858"
server_key_version: "encryptionkey@0"
I20260926 07:25:18.383082 5008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:18.385298 5022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:18.386515 5008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:18.386643 5008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "fc996d2b7d2f4ae3b8a8bd06bdb9a970"
format_stamp: "Formatted at 2026-09-26 07:25:18 on dist-test-slave-g04g"
server_key: "39849eec2df759b1b54a580ee344a77d"
server_key_iv: "899beb0002d7060cc329cbaeb4006858"
server_key_version: "encryptionkey@0"
I20260926 07:25:18.386746 5008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:18.391279 5008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:18.391857 5008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:18.392030 5008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:18.398321 5008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44347
I20260926 07:25:18.398342 5074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44347 every 8 connection(s)
I20260926 07:25:18.399230 5008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:18.402261 5075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:18.407720 5075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970: Bootstrap starting.
I20260926 07:25:18.409478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 5008
I20260926 07:25:18.409638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:18.409926 392 external_mini_cluster.cc:1428] Setting key 13aeb4c607dd739b9f607224c96e8d57
I20260926 07:25:18.410295 5075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:18.411245 5075 log.cc:824] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:18.413371 5075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970: No bootstrap required, opened a new log
I20260926 07:25:18.415591 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.411184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.0.98.62:44347)
0926 07:25:18.411932 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:18.411955 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:18.412002 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:25:18.412735 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:18.412753 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:18.413058 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:18.413355 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:18.414233 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:18.414238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:18.414250 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:18.414260 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:18.414317 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:18.414333 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:18.414431 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:18.414568 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 07:25:18.414833 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":154,"threads_started":1}
I20260926 07:25:18.416676 5075 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc996d2b7d2f4ae3b8a8bd06bdb9a970" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260926 07:25:18.416908 5075 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:18.416967 5075 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc996d2b7d2f4ae3b8a8bd06bdb9a970, State: Initialized, Role: FOLLOWER
I20260926 07:25:18.417418 5075 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc996d2b7d2f4ae3b8a8bd06bdb9a970" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260926 07:25:18.417572 5075 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:18.417623 5075 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:18.417712 5075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:18.418581 5075 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc996d2b7d2f4ae3b8a8bd06bdb9a970" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260926 07:25:18.418869 5075 leader_election.cc:302] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc996d2b7d2f4ae3b8a8bd06bdb9a970; no voters:
I20260926 07:25:18.419219 5075 leader_election.cc:288] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:18.419418 5080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:18.419745 5080 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [term 1 LEADER]: Becoming Leader. State: Replica: fc996d2b7d2f4ae3b8a8bd06bdb9a970, State: Running, Role: LEADER
I20260926 07:25:18.420195 5080 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc996d2b7d2f4ae3b8a8bd06bdb9a970" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260926 07:25:18.420547 5075 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:18.421948 5081 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc996d2b7d2f4ae3b8a8bd06bdb9a970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc996d2b7d2f4ae3b8a8bd06bdb9a970" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } } }
I20260926 07:25:18.421940 5082 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc996d2b7d2f4ae3b8a8bd06bdb9a970. Latest consensus state: current_term: 1 leader_uuid: "fc996d2b7d2f4ae3b8a8bd06bdb9a970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc996d2b7d2f4ae3b8a8bd06bdb9a970" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } } }
I20260926 07:25:18.422087 5082 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:18.422087 5081 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:18.422616 5092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:18.425562 5092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:18.430650 5092 catalog_manager.cc:1347] Generated new cluster ID: fbc35f2ef92649bb8761b737f630b9e3
I20260926 07:25:18.430719 5092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:18.439841 5092 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:18.440145 5092 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:18.448763 5092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970: Generated new TSK 0
I20260926 07:25:18.449544 5092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:25:18.457434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44347
--builtin_ntp_servers=127.0.98.20:37273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:25:18.573020 5099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:18.573266 5099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:18.573308 5099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:18.577859 5099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:18.577924 5099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:18.578022 5099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:18.583352 5099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:18.584424 5099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:18.585850 5099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:18.596627 5107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:18.596628 5104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:18.596735 5099 server_base.cc:1034] running on GCE node
W20260926 07:25:18.596645 5105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:18.597255 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:18.597955 5099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:18.599119 5099 hybrid_clock.cc:648] HybridClock initialized: now 1790407518599096 us; error 29 us; skew 500 ppm
I20260926 07:25:18.601063 5099 webserver.cc:460] Webserver started at http://127.0.98.1:35349/ using document root <none> and password file <none>
I20260926 07:25:18.601646 5099 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:18.601708 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:18.601922 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:18.604019 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "2975f9633ddd4bcc9777635e69b974d2"
format_stamp: "Formatted at 2026-09-26 07:25:18 on dist-test-slave-g04g"
server_key: "67efedf7df58934d6270b6b67ea1628b"
server_key_iv: "900590bb1c2b8a5e04fce024a83565de"
server_key_version: "encryptionkey@0"
I20260926 07:25:18.604574 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2975f9633ddd4bcc9777635e69b974d2"
format_stamp: "Formatted at 2026-09-26 07:25:18 on dist-test-slave-g04g"
server_key: "67efedf7df58934d6270b6b67ea1628b"
server_key_iv: "900590bb1c2b8a5e04fce024a83565de"
server_key_version: "encryptionkey@0"
I20260926 07:25:18.608124 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:18.610520 5113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:18.611806 5099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:18.611946 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "2975f9633ddd4bcc9777635e69b974d2"
format_stamp: "Formatted at 2026-09-26 07:25:18 on dist-test-slave-g04g"
server_key: "67efedf7df58934d6270b6b67ea1628b"
server_key_iv: "900590bb1c2b8a5e04fce024a83565de"
server_key_version: "encryptionkey@0"
I20260926 07:25:18.612056 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:18.616374 5099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:18.616925 5099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:18.617102 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:18.617616 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:18.618458 5099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:18.618528 5099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:18.618592 5099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:18.618640 5099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:18.629658 5099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42479
I20260926 07:25:18.629739 5226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42479 every 8 connection(s)
I20260926 07:25:18.630563 5099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:18.633751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 5099
I20260926 07:25:18.633877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:25:18.634183 392 external_mini_cluster.cc:1428] Setting key 4dc5c7ddf572b967485a9c9c548b48a1
I20260926 07:25:18.636272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44347
--builtin_ntp_servers=127.0.98.20:37273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 07:25:18.637566 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.632493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60989 (local address 127.0.98.62:44347)
0926 07:25:18.632661 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:18.632666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:18.633792 (+ 1126us) server_negotiation.cc:407] Connection header received
0926 07:25:18.635445 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:18.635449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:18.635490 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:18.635573 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:18.636939 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:18.636945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:18.636951 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:18.636967 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:18.637017 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:18.637030 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:18.637121 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:18.637329 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 07:25:18.637374 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:25:18.639073 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.632918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44347 (local address 127.0.98.1:60989)
0926 07:25:18.633603 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:18.633657 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:18.635238 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:18.635750 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:18.635758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:18.635878 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:18.636602 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:18.636606 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:18.636664 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:18.636754 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:18.637168 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:18.637179 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:18.637188 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:25:18.637836 (+ 648us) client_negotiation.cc:240] Negotiation successful
0926 07:25:18.638082 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260926 07:25:18.640357 5227 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44347
I20260926 07:25:18.640605 5227 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:18.640995 5227 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
I20260926 07:25:18.642505 5039 ts_manager.cc:194] Registered new tserver with Master: 2975f9633ddd4bcc9777635e69b974d2 (127.0.98.1:42479)
W20260926 07:25:18.751191 5230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:18.751461 5230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:18.751536 5230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:18.755525 5230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:18.755599 5230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:18.755687 5230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:18.760296 5230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:18.761298 5230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:18.762598 5230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:18.771687 5235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:18.771698 5236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:18.771678 5238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:18.772060 5230 server_base.cc:1034] running on GCE node
I20260926 07:25:18.772468 5230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:18.773129 5230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:18.774331 5230 hybrid_clock.cc:648] HybridClock initialized: now 1790407518774283 us; error 68 us; skew 500 ppm
I20260926 07:25:18.776340 5230 webserver.cc:460] Webserver started at http://127.0.98.2:38509/ using document root <none> and password file <none>
I20260926 07:25:18.776955 5230 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:18.777050 5230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:18.777282 5230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:18.779531 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "97e1f184492c4d5aa70548e8df825efd"
format_stamp: "Formatted at 2026-09-26 07:25:18 on dist-test-slave-g04g"
server_key: "37de0831509039eedf9cd2b1b43fc3fd"
server_key_iv: "996263232b3ad409e8cc2d3d77ba3014"
server_key_version: "encryptionkey@0"
I20260926 07:25:18.780123 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "97e1f184492c4d5aa70548e8df825efd"
format_stamp: "Formatted at 2026-09-26 07:25:18 on dist-test-slave-g04g"
server_key: "37de0831509039eedf9cd2b1b43fc3fd"
server_key_iv: "996263232b3ad409e8cc2d3d77ba3014"
server_key_version: "encryptionkey@0"
I20260926 07:25:18.783882 5230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:18.786306 5244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:18.787554 5230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:18.787688 5230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "97e1f184492c4d5aa70548e8df825efd"
format_stamp: "Formatted at 2026-09-26 07:25:18 on dist-test-slave-g04g"
server_key: "37de0831509039eedf9cd2b1b43fc3fd"
server_key_iv: "996263232b3ad409e8cc2d3d77ba3014"
server_key_version: "encryptionkey@0"
I20260926 07:25:18.787799 5230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:18.792721 5230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:18.793313 5230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:18.793496 5230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:18.794045 5230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:18.794945 5230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:18.795018 5230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:18.795116 5230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:18.795166 5230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:18.804105 5230 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39777
I20260926 07:25:18.804167 5357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39777 every 8 connection(s)
I20260926 07:25:18.805073 5230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:25:18.811520 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.806871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48749 (local address 127.0.98.62:44347)
0926 07:25:18.807019 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:18.807023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:18.808186 (+ 1163us) server_negotiation.cc:407] Connection header received
0926 07:25:18.809534 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:18.809537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:18.809573 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:18.809669 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:18.810987 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:18.810990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:18.810992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:18.811003 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:18.811024 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:18.811056 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:18.811182 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:18.811371 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:25:18.811410 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:25:18.812394 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.807343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44347 (local address 127.0.98.2:48749)
0926 07:25:18.808039 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:18.808081 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:18.809323 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:18.809889 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:18.809897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:18.809997 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:18.810726 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:18.810728 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:18.810766 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:18.810837 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:18.811182 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:18.811192 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:18.811200 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:25:18.811384 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 07:25:18.811578 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":114,"threads_started":1}
I20260926 07:25:18.813527 5358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44347
I20260926 07:25:18.813771 5358 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:18.814211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 5230
I20260926 07:25:18.814249 5358 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
I20260926 07:25:18.814388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:25:18.814759 392 external_mini_cluster.cc:1428] Setting key 1df4221b7aba13c4f5b6f89b9e15e9d7
I20260926 07:25:18.815470 5039 ts_manager.cc:194] Registered new tserver with Master: 97e1f184492c4d5aa70548e8df825efd (127.0.98.2:39777)
I20260926 07:25:18.816895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44347
--builtin_ntp_servers=127.0.98.20:37273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:25:18.928975 5361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:18.929219 5361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:18.929281 5361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:18.933374 5361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:18.933426 5361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:18.933509 5361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:18.938310 5361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:18.939379 5361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:18.940725 5361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:18.950421 5367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:18.950428 5366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:18.950562 5369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:18.950762 5361 server_base.cc:1034] running on GCE node
I20260926 07:25:18.951210 5361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:18.951874 5361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:18.953060 5361 hybrid_clock.cc:648] HybridClock initialized: now 1790407518953033 us; error 42 us; skew 500 ppm
I20260926 07:25:18.955331 5361 webserver.cc:460] Webserver started at http://127.0.98.3:41137/ using document root <none> and password file <none>
I20260926 07:25:18.955961 5361 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:18.956060 5361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:18.956321 5361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:18.958572 5361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "0e324d0c5d1943f2b021462337728d1b"
format_stamp: "Formatted at 2026-09-26 07:25:18 on dist-test-slave-g04g"
server_key: "0869c4ddef3aec0b3cc9a15cd76136bc"
server_key_iv: "52c4344044f401031626bfe169cf1b9d"
server_key_version: "encryptionkey@0"
I20260926 07:25:18.959244 5361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0e324d0c5d1943f2b021462337728d1b"
format_stamp: "Formatted at 2026-09-26 07:25:18 on dist-test-slave-g04g"
server_key: "0869c4ddef3aec0b3cc9a15cd76136bc"
server_key_iv: "52c4344044f401031626bfe169cf1b9d"
server_key_version: "encryptionkey@0"
I20260926 07:25:18.963265 5361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 07:25:18.966104 5375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:18.967406 5361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:18.967538 5361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "0e324d0c5d1943f2b021462337728d1b"
format_stamp: "Formatted at 2026-09-26 07:25:18 on dist-test-slave-g04g"
server_key: "0869c4ddef3aec0b3cc9a15cd76136bc"
server_key_iv: "52c4344044f401031626bfe169cf1b9d"
server_key_version: "encryptionkey@0"
I20260926 07:25:18.967648 5361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:18.972426 5361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:18.973007 5361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:18.973143 5361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:18.973603 5361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:18.974412 5361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:18.974463 5361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:18.974499 5361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:18.974522 5361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:18.983388 5361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35151
I20260926 07:25:18.983484 5488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35151 every 8 connection(s)
I20260926 07:25:18.984153 5361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:18.990077 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.985953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46609 (local address 127.0.98.62:44347)
0926 07:25:18.986120 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:18.986124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:18.987106 (+ 982us) server_negotiation.cc:407] Connection header received
0926 07:25:18.988416 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:18.988418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:18.988455 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:18.988532 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:18.989578 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:18.989585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:18.989589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:18.989608 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:18.989626 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:18.989634 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:18.989711 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:18.989928 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 07:25:18.989968 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 07:25:18.991183 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.986243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44347 (local address 127.0.98.3:46609)
0926 07:25:18.986928 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:18.986966 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:18.988211 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:18.988682 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:18.988688 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:18.988771 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:18.989300 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:18.989302 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:18.989347 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:18.989411 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:18.989746 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:18.989755 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:18.989762 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:25:18.989937 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 07:25:18.990182 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":135,"threads_started":1}
I20260926 07:25:18.992409 5489 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44347
I20260926 07:25:18.992655 5489 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:18.993060 5489 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
I20260926 07:25:18.993189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 5361
I20260926 07:25:18.993312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:18.993584 392 external_mini_cluster.cc:1428] Setting key 2243eef7c510c62116e38b76fd4b1c96
I20260926 07:25:18.994179 5038 ts_manager.cc:194] Registered new tserver with Master: 0e324d0c5d1943f2b021462337728d1b (127.0.98.3:35151)
I20260926 07:25:18.995698 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:25:18.999402 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:18.997610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.0.98.62:44347)
0926 07:25:18.997797 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:18.997801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:18.997945 (+ 144us) server_negotiation.cc:407] Connection header received
0926 07:25:18.998106 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:18.998110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:18.998149 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:18.998234 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:18.998942 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:18.998945 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:18.998948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:18.998966 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:18.998988 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:18.998999 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:18.999109 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:18.999182 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 07:25:18.999223 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 07:25:19.002264 5038 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:25:19.005347 5038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:19.017318 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.014015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42479 (local address 127.0.0.1:42114)
0926 07:25:19.014610 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:19.014659 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:19.014749 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:19.015750 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:19.015756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:19.015828 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:19.016468 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:19.016470 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:19.016489 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:19.016580 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:19.016957 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:19.016968 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:19.016977 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:25:19.017079 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:25:19.017141 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":76,"threads_started":1}
I20260926 07:25:19.017947 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.014128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.0.98.1:42479)
0926 07:25:19.015265 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:19.015284 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:19.015310 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:25:19.015366 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:19.015371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:19.015594 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:19.015739 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:19.016717 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:19.016723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:19.016734 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:19.016746 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:19.016788 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:19.016821 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:19.016913 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:19.017673 (+ 760us) server_negotiation.cc:299] Negotiation successful
0926 07:25:19.017743 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":120,"threads_started":1}
I20260926 07:25:19.018659 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.014716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39777 (local address 127.0.0.1:45930)
0926 07:25:19.016582 (+ 1866us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:19.016597 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:19.016689 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:19.017111 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:19.017117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:19.017133 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:19.017614 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:19.017616 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:19.017628 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:19.017662 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:19.018421 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:19.018423 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:19.018427 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:25:19.018488 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 07:25:19.018521 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1781,"thread_start_us":102,"threads_started":1}
I20260926 07:25:19.018903 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.014796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.0.98.3:35151)
0926 07:25:19.015464 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:19.015478 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:19.016681 (+ 1203us) server_negotiation.cc:407] Connection header received
0926 07:25:19.016735 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:19.016740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:19.016915 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:19.017044 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:19.018332 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:19.018341 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:19.018353 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:19.018368 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:19.018408 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:19.018437 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:19.018526 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:19.018687 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 07:25:19.018752 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":162,"threads_started":1}
I20260926 07:25:19.019205 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.014876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.0.98.2:39777)
0926 07:25:19.015332 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:19.015345 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:19.016777 (+ 1432us) server_negotiation.cc:407] Connection header received
0926 07:25:19.016825 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:19.016828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:19.016961 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:19.017359 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:19.018206 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:19.018212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:19.018224 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:19.018238 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:19.018292 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:19.018321 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:19.018918 (+ 597us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:19.018963 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 07:25:19.019024 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":101,"threads_started":1}
I20260926 07:25:19.020987 5159 tablet_service.cc:1467] Processing CreateTablet for tablet 31a3a46a7cdc42588b8350bc92ba28ca (DEFAULT_TABLE table=test-table [id=1759c88ec95041fa8e1b2924419e2dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:19.021703 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 31a3a46a7cdc42588b8350bc92ba28ca (DEFAULT_TABLE table=test-table [id=1759c88ec95041fa8e1b2924419e2dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:19.022190 5159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a3a46a7cdc42588b8350bc92ba28ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:19.022574 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a3a46a7cdc42588b8350bc92ba28ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:19.028316 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.014097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35151 (local address 127.0.0.1:33424)
0926 07:25:19.016487 (+ 2390us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:19.016500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:19.016580 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:19.017055 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:19.017060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:19.017078 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:19.018165 (+ 1087us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:19.018168 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:19.018179 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:19.018203 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:19.018562 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:19.018565 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:19.018568 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:25:19.018643 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:25:19.018673 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2327,"thread_start_us":105,"threads_started":1}
I20260926 07:25:19.029508 5506 tablet_bootstrap.cc:492] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2: Bootstrap starting.
I20260926 07:25:19.029974 5507 tablet_bootstrap.cc:492] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd: Bootstrap starting.
I20260926 07:25:19.031356 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 31a3a46a7cdc42588b8350bc92ba28ca (DEFAULT_TABLE table=test-table [id=1759c88ec95041fa8e1b2924419e2dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:19.031955 5507 tablet_bootstrap.cc:654] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:19.032223 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a3a46a7cdc42588b8350bc92ba28ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:19.032677 5507 log.cc:824] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:19.034639 5507 tablet_bootstrap.cc:492] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd: No bootstrap required, opened a new log
I20260926 07:25:19.034741 5506 tablet_bootstrap.cc:654] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:19.034845 5507 ts_tablet_manager.cc:1397] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 07:25:19.035580 5506 log.cc:824] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:19.037284 5506 tablet_bootstrap.cc:492] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2: No bootstrap required, opened a new log
I20260926 07:25:19.037480 5506 ts_tablet_manager.cc:1397] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260926 07:25:19.038942 5510 tablet_bootstrap.cc:492] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b: Bootstrap starting.
I20260926 07:25:19.039341 5507 raft_consensus.cc:348] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2975f9633ddd4bcc9777635e69b974d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42479 } } peers { permanent_uuid: "0e324d0c5d1943f2b021462337728d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } } peers { permanent_uuid: "97e1f184492c4d5aa70548e8df825efd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39777 } }
I20260926 07:25:19.039599 5507 raft_consensus.cc:374] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:19.039660 5507 raft_consensus.cc:729] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e1f184492c4d5aa70548e8df825efd, State: Initialized, Role: FOLLOWER
I20260926 07:25:19.040102 5507 consensus_queue.cc:260] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2975f9633ddd4bcc9777635e69b974d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42479 } } peers { permanent_uuid: "0e324d0c5d1943f2b021462337728d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } } peers { permanent_uuid: "97e1f184492c4d5aa70548e8df825efd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39777 } }
I20260926 07:25:19.040657 5358 heartbeater.cc:503] Master 127.0.98.62:44347 was elected leader, sending a full tablet report...
I20260926 07:25:19.041016 5507 ts_tablet_manager.cc:1428] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:25:19.041275 5510 tablet_bootstrap.cc:654] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:19.041448 5506 raft_consensus.cc:348] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2975f9633ddd4bcc9777635e69b974d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42479 } } peers { permanent_uuid: "0e324d0c5d1943f2b021462337728d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } } peers { permanent_uuid: "97e1f184492c4d5aa70548e8df825efd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39777 } }
I20260926 07:25:19.041693 5506 raft_consensus.cc:374] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:19.041761 5506 raft_consensus.cc:729] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2975f9633ddd4bcc9777635e69b974d2, State: Initialized, Role: FOLLOWER
I20260926 07:25:19.042052 5510 log.cc:824] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:19.042209 5506 consensus_queue.cc:260] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2975f9633ddd4bcc9777635e69b974d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42479 } } peers { permanent_uuid: "0e324d0c5d1943f2b021462337728d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } } peers { permanent_uuid: "97e1f184492c4d5aa70548e8df825efd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39777 } }
I20260926 07:25:19.043151 5506 ts_tablet_manager.cc:1428] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260926 07:25:19.043797 5510 tablet_bootstrap.cc:492] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b: No bootstrap required, opened a new log
I20260926 07:25:19.043990 5510 ts_tablet_manager.cc:1397] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:25:19.046993 5510 raft_consensus.cc:348] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2975f9633ddd4bcc9777635e69b974d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42479 } } peers { permanent_uuid: "0e324d0c5d1943f2b021462337728d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } } peers { permanent_uuid: "97e1f184492c4d5aa70548e8df825efd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39777 } }
I20260926 07:25:19.047210 5227 heartbeater.cc:503] Master 127.0.98.62:44347 was elected leader, sending a full tablet report...
I20260926 07:25:19.047277 5510 raft_consensus.cc:374] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:19.047322 5510 raft_consensus.cc:729] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e324d0c5d1943f2b021462337728d1b, State: Initialized, Role: FOLLOWER
I20260926 07:25:19.047714 5510 consensus_queue.cc:260] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2975f9633ddd4bcc9777635e69b974d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42479 } } peers { permanent_uuid: "0e324d0c5d1943f2b021462337728d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } } peers { permanent_uuid: "97e1f184492c4d5aa70548e8df825efd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39777 } }
I20260926 07:25:19.048514 5489 heartbeater.cc:503] Master 127.0.98.62:44347 was elected leader, sending a full tablet report...
I20260926 07:25:19.048627 5510 ts_tablet_manager.cc:1428] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 07:25:19.055980 5359 tablet.cc:2369] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:25:19.065033 5514 raft_consensus.cc:482] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:19.065241 5514 raft_consensus.cc:504] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2975f9633ddd4bcc9777635e69b974d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42479 } } peers { permanent_uuid: "0e324d0c5d1943f2b021462337728d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } } peers { permanent_uuid: "97e1f184492c4d5aa70548e8df825efd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39777 } }
I20260926 07:25:19.066658 5514 leader_election.cc:288] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2975f9633ddd4bcc9777635e69b974d2 (127.0.98.1:42479), 97e1f184492c4d5aa70548e8df825efd (127.0.98.2:39777)
I20260926 07:25:19.068660 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.066907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42479 (local address 127.0.98.3:48721)
0926 07:25:19.067074 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:19.067089 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:19.067169 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:19.067507 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:19.067513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:19.067534 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:19.068072 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:19.068074 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:19.068087 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:19.068121 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:19.068366 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:19.068369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:19.068372 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:25:19.068473 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 07:25:19.068514 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 07:25:19.069105 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.066989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48721 (local address 127.0.98.1:42479)
0926 07:25:19.067160 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:19.067166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:19.067184 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:25:19.067255 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:19.067258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:19.067302 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:19.067402 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:19.068222 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:19.068226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:19.068228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:19.068240 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:19.068261 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:19.068271 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:19.068366 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:19.068915 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 07:25:19.068958 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:25:19.069448 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.067374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44257 (local address 127.0.98.2:39777)
0926 07:25:19.067493 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:19.067497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:19.067605 (+ 108us) server_negotiation.cc:407] Connection header received
0926 07:25:19.068150 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:19.068156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:19.068201 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:19.068249 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:19.069071 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:19.069074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:19.069076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:19.069087 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:19.069106 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:19.069114 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:19.069218 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:19.069303 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 07:25:19.069349 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:25:19.069700 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a3a46a7cdc42588b8350bc92ba28ca" candidate_uuid: "0e324d0c5d1943f2b021462337728d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2975f9633ddd4bcc9777635e69b974d2" is_pre_election: true
I20260926 07:25:19.069984 5181 raft_consensus.cc:2454] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e324d0c5d1943f2b021462337728d1b in term 0.
I20260926 07:25:19.070329 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.067236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39777 (local address 127.0.98.3:44257)
0926 07:25:19.067520 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:19.067532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:19.067665 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:19.068376 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:19.068379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:19.068395 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:19.068928 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:19.068930 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:19.068940 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:19.068963 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:19.069213 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:19.069216 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:19.069219 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:25:19.069305 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 07:25:19.069337 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":87,"threads_started":1}
I20260926 07:25:19.070463 5379 leader_election.cc:302] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e324d0c5d1943f2b021462337728d1b, 2975f9633ddd4bcc9777635e69b974d2; no voters:
I20260926 07:25:19.070757 5514 raft_consensus.cc:2781] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:19.070858 5514 raft_consensus.cc:482] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:19.070915 5514 raft_consensus.cc:3037] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:19.071244 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a3a46a7cdc42588b8350bc92ba28ca" candidate_uuid: "0e324d0c5d1943f2b021462337728d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e1f184492c4d5aa70548e8df825efd" is_pre_election: true
I20260926 07:25:19.071547 5312 raft_consensus.cc:2454] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e324d0c5d1943f2b021462337728d1b in term 0.
I20260926 07:25:19.072191 5514 raft_consensus.cc:504] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2975f9633ddd4bcc9777635e69b974d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42479 } } peers { permanent_uuid: "0e324d0c5d1943f2b021462337728d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } } peers { permanent_uuid: "97e1f184492c4d5aa70548e8df825efd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39777 } }
I20260926 07:25:19.072499 5514 leader_election.cc:288] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [CANDIDATE]: Term 1 election: Requested vote from peers 2975f9633ddd4bcc9777635e69b974d2 (127.0.98.1:42479), 97e1f184492c4d5aa70548e8df825efd (127.0.98.2:39777)
I20260926 07:25:19.072893 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a3a46a7cdc42588b8350bc92ba28ca" candidate_uuid: "0e324d0c5d1943f2b021462337728d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e1f184492c4d5aa70548e8df825efd"
I20260926 07:25:19.072887 5181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a3a46a7cdc42588b8350bc92ba28ca" candidate_uuid: "0e324d0c5d1943f2b021462337728d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2975f9633ddd4bcc9777635e69b974d2"
I20260926 07:25:19.073004 5181 raft_consensus.cc:3037] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:19.073004 5312 raft_consensus.cc:3037] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:19.074268 5181 raft_consensus.cc:2454] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e324d0c5d1943f2b021462337728d1b in term 1.
I20260926 07:25:19.074271 5312 raft_consensus.cc:2454] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e324d0c5d1943f2b021462337728d1b in term 1.
I20260926 07:25:19.074554 5379 leader_election.cc:302] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e324d0c5d1943f2b021462337728d1b, 2975f9633ddd4bcc9777635e69b974d2; no voters:
I20260926 07:25:19.074769 5514 raft_consensus.cc:2781] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:19.075001 5514 raft_consensus.cc:686] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [term 1 LEADER]: Becoming Leader. State: Replica: 0e324d0c5d1943f2b021462337728d1b, State: Running, Role: LEADER
I20260926 07:25:19.075330 5514 consensus_queue.cc:237] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2975f9633ddd4bcc9777635e69b974d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42479 } } peers { permanent_uuid: "0e324d0c5d1943f2b021462337728d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } } peers { permanent_uuid: "97e1f184492c4d5aa70548e8df825efd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39777 } }
I20260926 07:25:19.078459 5039 catalog_manager.cc:5495] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e324d0c5d1943f2b021462337728d1b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0e324d0c5d1943f2b021462337728d1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2975f9633ddd4bcc9777635e69b974d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e324d0c5d1943f2b021462337728d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97e1f184492c4d5aa70548e8df825efd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39777 } health_report { overall_health: UNKNOWN } } }
I20260926 07:25:19.095283 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.093847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.0.98.3:35151)
0926 07:25:19.093980 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:19.093984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:19.093997 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:25:19.094134 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:19.094137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:19.094204 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:19.094277 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:19.094941 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:19.094944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:19.094946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:19.094956 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:19.094974 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:19.094982 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:19.095126 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:19.095156 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 07:25:19.095187 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:25:19.100754 5312 raft_consensus.cc:1261] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd [term 1 FOLLOWER]: Refusing update from remote peer 0e324d0c5d1943f2b021462337728d1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:19.100795 5181 raft_consensus.cc:1261] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2 [term 1 FOLLOWER]: Refusing update from remote peer 0e324d0c5d1943f2b021462337728d1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:19.101464 5516 consensus_queue.cc:1035] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e1f184492c4d5aa70548e8df825efd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:19.101668 5514 consensus_queue.cc:1035] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2975f9633ddd4bcc9777635e69b974d2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:19.109130 5521 mvcc.cc:205] Tried to move back new op lower bound from 7333509198230245376 to 7333509198135275520. Current Snapshot: MvccSnapshot[applied={T|T < 7333509198230245376}]
I20260926 07:25:19.110647 5520 mvcc.cc:205] Tried to move back new op lower bound from 7333509198230245376 to 7333509198135275520. Current Snapshot: MvccSnapshot[applied={T|T < 7333509198230245376}]
I20260926 07:25:19.118045 5037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:19.118225 5037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:19.120740 5037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc996d2b7d2f4ae3b8a8bd06bdb9a970: Sending DeleteTablet for 3 replicas of tablet 31a3a46a7cdc42588b8350bc92ba28ca
I20260926 07:25:19.121475 5159 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a3a46a7cdc42588b8350bc92ba28ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:19 UTC) from {username='slave'} at 127.0.0.1:42114
I20260926 07:25:19.121583 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a3a46a7cdc42588b8350bc92ba28ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:19 UTC) from {username='slave'} at 127.0.0.1:33424
I20260926 07:25:19.121770 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a3a46a7cdc42588b8350bc92ba28ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:19 UTC) from {username='slave'} at 127.0.0.1:45930
I20260926 07:25:19.122219 5531 tablet_replica.cc:330] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd: stopping tablet replica
I20260926 07:25:19.122413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 5099
I20260926 07:25:19.122437 5529 tablet_replica.cc:330] T 31a3a46a7cdc42588b8350bc92ba28ca P 2975f9633ddd4bcc9777635e69b974d2: stopping tablet replica
I20260926 07:25:19.123858 5531 raft_consensus.cc:2229] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:19.124202 5531 raft_consensus.cc:2258] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:19.125748 5531 ts_tablet_manager.cc:1905] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:19.130055 5531 ts_tablet_manager.cc:1918] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:19.130162 5531 log.cc:1196] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/31a3a46a7cdc42588b8350bc92ba28ca
I20260926 07:25:19.130420 5530 tablet_replica.cc:330] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b: stopping tablet replica
I20260926 07:25:19.130519 5531 ts_tablet_manager.cc:1939] T 31a3a46a7cdc42588b8350bc92ba28ca P 97e1f184492c4d5aa70548e8df825efd: Deleting consensus metadata
I20260926 07:25:19.130790 5530 raft_consensus.cc:2229] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [term 1 LEADER]: Raft consensus shutting down.
W20260926 07:25:19.130925 5026 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42479 (error 108)
I20260926 07:25:19.131289 5530 raft_consensus.cc:2258] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:19.132141 5026 catalog_manager.cc:4841] TS 97e1f184492c4d5aa70548e8df825efd (127.0.98.2:39777): tablet 31a3a46a7cdc42588b8350bc92ba28ca (table test-table [id=1759c88ec95041fa8e1b2924419e2dcb]) successfully deleted
I20260926 07:25:19.132513 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.132000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42479 (local address 127.0.0.1:42120)
0926 07:25:19.132386 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:19.132446 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306}
W20260926 07:25:19.132682 5026 catalog_manager.cc:4584] TS 2975f9633ddd4bcc9777635e69b974d2 (127.0.98.1:42479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31a3a46a7cdc42588b8350bc92ba28ca: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42479: connect: Connection refused (error 111)
I20260926 07:25:19.132859 5530 ts_tablet_manager.cc:1905] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:19.133062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 5230
I20260926 07:25:19.135370 5530 ts_tablet_manager.cc:1918] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:19.135459 5530 log.cc:1196] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/31a3a46a7cdc42588b8350bc92ba28ca
I20260926 07:25:19.135761 5530 ts_tablet_manager.cc:1939] T 31a3a46a7cdc42588b8350bc92ba28ca P 0e324d0c5d1943f2b021462337728d1b: Deleting consensus metadata
I20260926 07:25:19.136816 5023 catalog_manager.cc:4841] TS 0e324d0c5d1943f2b021462337728d1b (127.0.98.3:35151): tablet 31a3a46a7cdc42588b8350bc92ba28ca (table test-table [id=1759c88ec95041fa8e1b2924419e2dcb]) successfully deleted
I20260926 07:25:19.139856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 5361
I20260926 07:25:19.145735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 5008
2026-09-26T07:25:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (930 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 07:25:19.167173 392 mini_oidc.cc:152] Starting JWKS server
I20260926 07:25:19.168753 392 webserver.cc:460] Webserver started at https://127.0.0.1:38511/ using document root <none> and password file <none>
I20260926 07:25:19.168881 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:25:19.169091 392 webserver.cc:460] Webserver started at http://127.0.0.1:44899/ using document root <none> and password file <none>
2026-09-26T07:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:19Z Disabled control of system clock
I20260926 07:25:19.177650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34329
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34329
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38511/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/testchainca.cert with env {}
W20260926 07:25:19.285113 5538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:19.285372 5538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:19.285410 5538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:19.289350 5538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:19.289413 5538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:19.289428 5538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:19.289445 5538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:19.289460 5538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:25:19.289476 5538 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:25:19.289489 5538 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38511/jwks.json
I20260926 07:25:19.294729 5538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34329
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38511/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:19.295696 5538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:19.296633 5538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:19.304801 5543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:19.304828 5546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:19.304812 5544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:19.305035 5538 server_base.cc:1034] running on GCE node
I20260926 07:25:19.305652 5538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:19.306632 5538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:19.307778 5538 hybrid_clock.cc:648] HybridClock initialized: now 1790407519307754 us; error 49 us; skew 500 ppm
I20260926 07:25:19.309489 5538 webserver.cc:460] Webserver started at http://127.0.98.62:43445/ using document root <none> and password file <none>
I20260926 07:25:19.309998 5538 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:19.310055 5538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:19.310199 5538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:19.312244 5538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "6bff66b2b9b542f8857c214929031f8b"
format_stamp: "Formatted at 2026-09-26 07:25:19 on dist-test-slave-g04g"
server_key: "4c60c4678313143b3bb123cf0266a053"
server_key_iv: "c9faf3be94dbd95715f4bf0559975384"
server_key_version: "encryptionkey@0"
I20260926 07:25:19.312755 5538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "6bff66b2b9b542f8857c214929031f8b"
format_stamp: "Formatted at 2026-09-26 07:25:19 on dist-test-slave-g04g"
server_key: "4c60c4678313143b3bb123cf0266a053"
server_key_iv: "c9faf3be94dbd95715f4bf0559975384"
server_key_version: "encryptionkey@0"
I20260926 07:25:19.316322 5538 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:19.318598 5552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:19.319921 5538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:19.320029 5538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "6bff66b2b9b542f8857c214929031f8b"
format_stamp: "Formatted at 2026-09-26 07:25:19 on dist-test-slave-g04g"
server_key: "4c60c4678313143b3bb123cf0266a053"
server_key_iv: "c9faf3be94dbd95715f4bf0559975384"
server_key_version: "encryptionkey@0"
I20260926 07:25:19.320127 5538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:19.334928 5538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:19.335590 5538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:19.335745 5538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:19.343149 5538 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34329
I20260926 07:25:19.343132 5604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34329 every 8 connection(s)
I20260926 07:25:19.344126 5538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:19.347297 5605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:19.352685 5605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b: Bootstrap starting.
I20260926 07:25:19.353087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 5538
I20260926 07:25:19.353202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:19.353469 392 external_mini_cluster.cc:1428] Setting key 664aee4da9393e11119b09e5284c8a79
I20260926 07:25:19.355255 5605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:19.355969 5605 log.cc:824] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:19.357846 5605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b: No bootstrap required, opened a new log
I20260926 07:25:19.362023 5605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bff66b2b9b542f8857c214929031f8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260926 07:25:19.362257 5605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:19.362335 5605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bff66b2b9b542f8857c214929031f8b, State: Initialized, Role: FOLLOWER
I20260926 07:25:19.362490 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.354735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.0.98.62:34329)
0926 07:25:19.355405 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:19.355429 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:19.355463 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:25:19.356061 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:19.356103 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:19.356501 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:19.356766 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:19.357089 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.358081 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.359267 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.359751 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.359875 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:19.360685 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:19.360690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:19.360702 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:19.361081 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:19.361133 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:19.361151 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:19.361226 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:19.361406 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 07:25:19.361693 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":135,"threads_started":1}
I20260926 07:25:19.362830 5605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bff66b2b9b542f8857c214929031f8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260926 07:25:19.362977 5605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:19.363062 5605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:19.363158 5605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:19.364207 5605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bff66b2b9b542f8857c214929031f8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260926 07:25:19.364557 5605 leader_election.cc:302] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bff66b2b9b542f8857c214929031f8b; no voters:
I20260926 07:25:19.364889 5605 leader_election.cc:288] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:19.364976 5610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:19.365178 5610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [term 1 LEADER]: Becoming Leader. State: Replica: 6bff66b2b9b542f8857c214929031f8b, State: Running, Role: LEADER
I20260926 07:25:19.365469 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:25:19.365475 5610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bff66b2b9b542f8857c214929031f8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260926 07:25:19.365998 5605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:19.366952 5611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bff66b2b9b542f8857c214929031f8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bff66b2b9b542f8857c214929031f8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } }
I20260926 07:25:19.367079 5611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:19.367358 5612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bff66b2b9b542f8857c214929031f8b. Latest consensus state: current_term: 1 leader_uuid: "6bff66b2b9b542f8857c214929031f8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bff66b2b9b542f8857c214929031f8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } }
I20260926 07:25:19.367476 5612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:19.367753 5619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:19.370441 5619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:19.375469 5619 catalog_manager.cc:1347] Generated new cluster ID: 54fd8ad0dc68415ab2ddd1ef563225e4
I20260926 07:25:19.375550 5619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:19.388070 5619 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:19.389585 5619 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:19.398439 5619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bff66b2b9b542f8857c214929031f8b: Generated new TSK 0
I20260926 07:25:19.399261 5619 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 07:25:19.437896 5635 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 07:25:19.438107 5635 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 07:25:19.438601 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.418918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.0.98.62:34329)
0926 07:25:19.419110 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:19.419114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:19.419284 (+ 170us) server_negotiation.cc:407] Connection header received
0926 07:25:19.419483 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:19.419487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:19.419544 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:19.419657 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:25:19.420069 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.421050 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.422306 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.422740 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.422879 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:19.423322 (+ 443us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:25:19.438217 (+ 14895us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38511/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 07:25:19.438494 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57884: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38511/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 07:25:19.438699 5608 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57884: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38511/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 07:25:19.440078 5634 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:19.418860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.0.1:57884)
0926 07:25:19.419199 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:19.419217 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:19.419332 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:19.419777 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:19.419781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:19.419795 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:25:19.419922 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:19.419929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.421171 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.421175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:19.422179 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:19.422189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.422863 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.422909 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:19.423026 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:19.423117 (+ 91us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:25:19.438474 (+ 15357us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38511/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 07:25:19.438722 (+ 248us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34329: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38511/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":110,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:34329: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38511/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 07:25:19.440882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 5538
2026-09-26T07:25:19Z chronyd exiting
I20260926 07:25:19.477943 392 test_util.cc:175] -----------------------------------------------
I20260926 07:25:19.478027 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790407489657243-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 07:25:19.478981 392 mini_oidc.cc:152] Starting JWKS server
I20260926 07:25:19.479866 392 webserver.cc:460] Webserver started at https://127.0.0.1:46413/ using document root <none> and password file <none>
I20260926 07:25:19.479944 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:25:19.480149 392 webserver.cc:460] Webserver started at http://127.0.0.1:43093/ using document root <none> and password file <none>
2026-09-26T07:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:19Z Disabled control of system clock
I20260926 07:25:19.489511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44775
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44775
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46413/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/testchainca.cert with env {}
W20260926 07:25:19.604598 5641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:19.604836 5641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:19.604874 5641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:19.608739 5641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:19.608788 5641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:19.608803 5641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:19.608819 5641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:19.608834 5641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:25:19.608850 5641 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:25:19.608863 5641 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46413/jwks.json
I20260926 07:25:19.614053 5641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46413/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:19.615056 5641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:19.615949 5641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:19.623925 5649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:19.623931 5647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:19.623970 5646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:19.624307 5641 server_base.cc:1034] running on GCE node
I20260926 07:25:19.624809 5641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:19.625834 5641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:19.627017 5641 hybrid_clock.cc:648] HybridClock initialized: now 1790407519627002 us; error 26 us; skew 500 ppm
I20260926 07:25:19.628904 5641 webserver.cc:460] Webserver started at http://127.0.98.62:42683/ using document root <none> and password file <none>
I20260926 07:25:19.629462 5641 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:19.629549 5641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:19.629752 5641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:19.631810 5641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "abcc70b6a8144c4293b6242ae12f7643"
format_stamp: "Formatted at 2026-09-26 07:25:19 on dist-test-slave-g04g"
server_key: "5e3b47aad6eb13f65806cd28378918c4"
server_key_iv: "1aa294e6d1aa43411cc79eedf2c5c733"
server_key_version: "encryptionkey@0"
I20260926 07:25:19.632362 5641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "abcc70b6a8144c4293b6242ae12f7643"
format_stamp: "Formatted at 2026-09-26 07:25:19 on dist-test-slave-g04g"
server_key: "5e3b47aad6eb13f65806cd28378918c4"
server_key_iv: "1aa294e6d1aa43411cc79eedf2c5c733"
server_key_version: "encryptionkey@0"
I20260926 07:25:19.635883 5641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:25:19.638191 5655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:19.639395 5641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 07:25:19.639530 5641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "abcc70b6a8144c4293b6242ae12f7643"
format_stamp: "Formatted at 2026-09-26 07:25:19 on dist-test-slave-g04g"
server_key: "5e3b47aad6eb13f65806cd28378918c4"
server_key_iv: "1aa294e6d1aa43411cc79eedf2c5c733"
server_key_version: "encryptionkey@0"
I20260926 07:25:19.639639 5641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:19.655103 5641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:19.655735 5641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:19.655937 5641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:19.662534 5641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44775
I20260926 07:25:19.662545 5707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44775 every 8 connection(s)
I20260926 07:25:19.663483 5641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:19.665267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 5641
I20260926 07:25:19.665398 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:19.665680 392 external_mini_cluster.cc:1428] Setting key 74116d80fcc139dc722ce7021da332ee
I20260926 07:25:19.666882 5708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:19.673671 5708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643: Bootstrap starting.
I20260926 07:25:19.676090 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.667077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.0.98.62:44775)
0926 07:25:19.667736 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:19.667755 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:19.667799 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:25:19.668667 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:19.668690 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:19.669186 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:19.669560 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:19.670006 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.671476 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.672338 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.673048 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.673205 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:19.674078 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:19.674085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:19.674100 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:19.674576 (+ 476us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:19.674642 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:19.674666 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:19.674767 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:19.674884 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 07:25:19.675197 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260926 07:25:19.676414 5708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:19.677261 5708 log.cc:824] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:19.678941 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:25:19.679380 5708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643: No bootstrap required, opened a new log
I20260926 07:25:19.682844 5708 raft_consensus.cc:348] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcc70b6a8144c4293b6242ae12f7643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44775 } }
I20260926 07:25:19.683101 5708 raft_consensus.cc:374] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:19.683208 5708 raft_consensus.cc:729] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abcc70b6a8144c4293b6242ae12f7643, State: Initialized, Role: FOLLOWER
I20260926 07:25:19.683660 5708 consensus_queue.cc:260] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcc70b6a8144c4293b6242ae12f7643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44775 } }
I20260926 07:25:19.683787 5708 raft_consensus.cc:388] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:19.683862 5708 raft_consensus.cc:482] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:19.683965 5708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:19.684962 5708 raft_consensus.cc:504] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcc70b6a8144c4293b6242ae12f7643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44775 } }
I20260926 07:25:19.685292 5708 leader_election.cc:302] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abcc70b6a8144c4293b6242ae12f7643; no voters:
I20260926 07:25:19.685628 5708 leader_election.cc:288] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:19.685690 5713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:19.685941 5713 raft_consensus.cc:686] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [term 1 LEADER]: Becoming Leader. State: Replica: abcc70b6a8144c4293b6242ae12f7643, State: Running, Role: LEADER
I20260926 07:25:19.686231 5713 consensus_queue.cc:237] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcc70b6a8144c4293b6242ae12f7643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44775 } }
I20260926 07:25:19.686803 5708 sys_catalog.cc:564] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:19.687825 5714 sys_catalog.cc:455] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abcc70b6a8144c4293b6242ae12f7643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcc70b6a8144c4293b6242ae12f7643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44775 } } }
I20260926 07:25:19.687971 5714 sys_catalog.cc:458] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:19.688324 5722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:19.688309 5715 sys_catalog.cc:455] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abcc70b6a8144c4293b6242ae12f7643. Latest consensus state: current_term: 1 leader_uuid: "abcc70b6a8144c4293b6242ae12f7643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abcc70b6a8144c4293b6242ae12f7643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44775 } } }
I20260926 07:25:19.688934 5715 sys_catalog.cc:458] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:19.691480 5722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:19.696866 5722 catalog_manager.cc:1347] Generated new cluster ID: 950f296338ca4d808cf88ba28564bed2
I20260926 07:25:19.696928 5722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:19.706411 5722 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:19.707607 5722 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:19.717155 5722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abcc70b6a8144c4293b6242ae12f7643: Generated new TSK 0
I20260926 07:25:19.717932 5722 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 07:25:19.742683 5738 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 07:25:19.742767 5738 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 07:25:19.743431 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.727821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.98.62:44775)
0926 07:25:19.727964 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:19.727969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:19.728165 (+ 196us) server_negotiation.cc:407] Connection header received
0926 07:25:19.728337 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:19.728339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:19.728384 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:19.728472 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:25:19.728712 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.729216 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.730422 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.730630 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.730700 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:19.731007 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:25:19.743098 (+ 12091us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46413/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 07:25:19.743328 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46413/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 07:25:19.743507 5711 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46413/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 07:25:19.743626 5737 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:19.727759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44775 (local address 127.0.0.1:48348)
0926 07:25:19.728070 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:19.728085 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:19.728203 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:19.728493 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:19.728496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:19.728505 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:25:19.728613 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:19.728619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.729319 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.729323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:19.730305 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:19.730314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.730718 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.730736 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:19.730824 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:19.730852 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:25:19.743292 (+ 12440us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46413/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 07:25:19.743456 (+ 164us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44775: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46413/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
I20260926 07:25:19.744480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 5641
2026-09-26T07:25:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 07:25:19.783766 392 mini_oidc.cc:152] Starting JWKS server
I20260926 07:25:19.784622 392 webserver.cc:460] Webserver started at https://127.0.0.1:45969/ using document root <none> and password file <none>
I20260926 07:25:19.784751 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:25:19.785017 392 webserver.cc:460] Webserver started at http://127.0.0.1:43067/ using document root <none> and password file <none>
2026-09-26T07:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:19Z Disabled control of system clock
I20260926 07:25:19.793779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34773
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34773
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45969/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 07:25:19.899956 5744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:19.900213 5744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:19.900250 5744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:19.904278 5744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:19.904330 5744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:19.904345 5744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:19.904361 5744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:19.904395 5744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:25:19.904431 5744 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:25:19.904451 5744 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45969/jwks.json
I20260926 07:25:19.909094 5744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34773
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45969/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:19.910094 5744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:19.911459 5744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:19.919597 5750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:19.919581 5749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:19.919806 5744 server_base.cc:1034] running on GCE node
W20260926 07:25:19.919844 5752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:19.920433 5744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:19.921478 5744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:19.922672 5744 hybrid_clock.cc:648] HybridClock initialized: now 1790407519922660 us; error 28 us; skew 500 ppm
I20260926 07:25:19.924479 5744 webserver.cc:460] Webserver started at http://127.0.98.62:45429/ using document root <none> and password file <none>
I20260926 07:25:19.925033 5744 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:19.925122 5744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:19.925345 5744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:19.927357 5744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "080cca7d1fed40a68ac8fbce9b886427"
format_stamp: "Formatted at 2026-09-26 07:25:19 on dist-test-slave-g04g"
server_key: "3d3bbd50eb9374b21bfca0bec19b4edb"
server_key_iv: "470c8d5fecc8f7fbad5ac3b71e321b89"
server_key_version: "encryptionkey@0"
I20260926 07:25:19.927930 5744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "080cca7d1fed40a68ac8fbce9b886427"
format_stamp: "Formatted at 2026-09-26 07:25:19 on dist-test-slave-g04g"
server_key: "3d3bbd50eb9374b21bfca0bec19b4edb"
server_key_iv: "470c8d5fecc8f7fbad5ac3b71e321b89"
server_key_version: "encryptionkey@0"
I20260926 07:25:19.931401 5744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:19.933647 5758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:19.934820 5744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:19.934952 5744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "080cca7d1fed40a68ac8fbce9b886427"
format_stamp: "Formatted at 2026-09-26 07:25:19 on dist-test-slave-g04g"
server_key: "3d3bbd50eb9374b21bfca0bec19b4edb"
server_key_iv: "470c8d5fecc8f7fbad5ac3b71e321b89"
server_key_version: "encryptionkey@0"
I20260926 07:25:19.935084 5744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:19.958518 5744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:19.959335 5744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:19.959535 5744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:19.966918 5744 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34773
I20260926 07:25:19.966925 5810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34773 every 8 connection(s)
I20260926 07:25:19.967912 5744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:19.969060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 5744
I20260926 07:25:19.969151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:19.969444 392 external_mini_cluster.cc:1428] Setting key 1711977ac1b95e9831d68a94ebb164f1
I20260926 07:25:19.971580 5811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:19.978340 5811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427: Bootstrap starting.
I20260926 07:25:19.979672 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:19.970692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.0.98.62:34773)
0926 07:25:19.971359 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:19.971378 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:19.971419 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:25:19.972263 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:19.972286 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:19.972760 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:19.973125 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:19.973553 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.975023 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.975957 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:19.976642 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:19.976812 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:19.977643 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:19.977650 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:19.977666 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:19.978155 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:19.978224 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:19.978248 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:19.978345 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:19.978477 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 07:25:19.978731 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":104,"threads_started":1}
I20260926 07:25:19.980901 5811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:19.981756 5811 log.cc:824] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:19.982615 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:25:19.983959 5811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427: No bootstrap required, opened a new log
I20260926 07:25:19.988026 5811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cca7d1fed40a68ac8fbce9b886427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34773 } }
I20260926 07:25:19.988255 5811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:19.988291 5811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080cca7d1fed40a68ac8fbce9b886427, State: Initialized, Role: FOLLOWER
I20260926 07:25:19.988663 5811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cca7d1fed40a68ac8fbce9b886427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34773 } }
I20260926 07:25:19.988816 5811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:19.988895 5811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:19.989015 5811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:19.990020 5811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cca7d1fed40a68ac8fbce9b886427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34773 } }
I20260926 07:25:19.990315 5811 leader_election.cc:302] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 080cca7d1fed40a68ac8fbce9b886427; no voters:
I20260926 07:25:19.990563 5811 leader_election.cc:288] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:19.990664 5816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:19.990926 5816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [term 1 LEADER]: Becoming Leader. State: Replica: 080cca7d1fed40a68ac8fbce9b886427, State: Running, Role: LEADER
I20260926 07:25:19.991256 5816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cca7d1fed40a68ac8fbce9b886427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34773 } }
I20260926 07:25:19.991861 5811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:19.992986 5818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 080cca7d1fed40a68ac8fbce9b886427. Latest consensus state: current_term: 1 leader_uuid: "080cca7d1fed40a68ac8fbce9b886427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cca7d1fed40a68ac8fbce9b886427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34773 } } }
I20260926 07:25:19.992986 5817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "080cca7d1fed40a68ac8fbce9b886427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080cca7d1fed40a68ac8fbce9b886427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34773 } } }
I20260926 07:25:19.993129 5818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:19.993129 5817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:19.993662 5825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:19.996766 5825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:20.002396 5825 catalog_manager.cc:1347] Generated new cluster ID: 54c5e5549d8c4473906cf1122613c664
I20260926 07:25:20.002468 5825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:20.033424 5825 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:20.034374 5825 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:20.041628 5825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 080cca7d1fed40a68ac8fbce9b886427: Generated new TSK 0
I20260926 07:25:20.042326 5825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:25:20.061268 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:20.056458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.0.98.62:34773)
0926 07:25:20.056601 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:20.056606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:20.056799 (+ 193us) server_negotiation.cc:407] Connection header received
0926 07:25:20.056978 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:20.056980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:20.057026 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:20.057108 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:25:20.057368 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:20.057852 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:20.059117 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:20.059351 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:20.059428 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:20.059738 (+ 310us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:25:20.060879 (+ 1141us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 07:25:20.061114 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 07:25:20.061336 5814 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 07:25:20.061354 5840 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:20.056374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34773 (local address 127.0.0.1:48182)
0926 07:25:20.056704 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:20.056719 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:20.056844 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:20.057139 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:20.057143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:20.057151 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:25:20.057265 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:20.057271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:20.057986 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:20.057989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:20.058983 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:20.058993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:20.059442 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:20.059464 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:20.059549 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:20.059576 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:25:20.061053 (+ 1477us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 07:25:20.061182 (+ 129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34773: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":112,"threads_started":1}
I20260926 07:25:20.061868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 5744
E20260926 07:25:20.074234 5841 webserver.cc:533] Webserver: sslize failed:
E20260926 07:25:20.074297 5841 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T07:25:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (318 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T07:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:20Z Disabled control of system clock
I20260926 07:25:20.110502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42291
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42291
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/testchainca.cert with env {}
W20260926 07:25:20.218794 5845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:20.219082 5845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:20.219146 5845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:20.222942 5845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:20.222990 5845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:20.223004 5845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:20.223021 5845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:20.223096 5845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:25:20.223119 5845 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:25:20.223145 5845 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 07:25:20.228318 5845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42291
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:20.229203 5845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:20.230068 5845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:20.238041 5853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:20.238092 5851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:20.238106 5850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:20.238291 5845 server_base.cc:1034] running on GCE node
I20260926 07:25:20.238849 5845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:20.239931 5845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:20.241114 5845 hybrid_clock.cc:648] HybridClock initialized: now 1790407520241100 us; error 27 us; skew 500 ppm
I20260926 07:25:20.242879 5845 webserver.cc:460] Webserver started at http://127.0.98.62:40509/ using document root <none> and password file <none>
I20260926 07:25:20.243484 5845 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:20.243574 5845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:20.243798 5845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:20.245782 5845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "6612c449329b41a4a90c95c01888abd5"
format_stamp: "Formatted at 2026-09-26 07:25:20 on dist-test-slave-g04g"
server_key: "0279ff6c9bcbab4f11ea4a8fc9d2c794"
server_key_iv: "c595a15847218b5c66afabb88cb88403"
server_key_version: "encryptionkey@0"
I20260926 07:25:20.246315 5845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "6612c449329b41a4a90c95c01888abd5"
format_stamp: "Formatted at 2026-09-26 07:25:20 on dist-test-slave-g04g"
server_key: "0279ff6c9bcbab4f11ea4a8fc9d2c794"
server_key_iv: "c595a15847218b5c66afabb88cb88403"
server_key_version: "encryptionkey@0"
I20260926 07:25:20.249727 5845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:25:20.251956 5859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:20.253149 5845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:20.253310 5845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "6612c449329b41a4a90c95c01888abd5"
format_stamp: "Formatted at 2026-09-26 07:25:20 on dist-test-slave-g04g"
server_key: "0279ff6c9bcbab4f11ea4a8fc9d2c794"
server_key_iv: "c595a15847218b5c66afabb88cb88403"
server_key_version: "encryptionkey@0"
I20260926 07:25:20.253417 5845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:20.275727 5845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:20.276355 5845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:20.276541 5845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:20.283535 5845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42291
I20260926 07:25:20.283543 5911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42291 every 8 connection(s)
I20260926 07:25:20.284428 5845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:20.285835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 5845
I20260926 07:25:20.285967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:20.286365 392 external_mini_cluster.cc:1428] Setting key 2853d546b1e181653bc060a5e3f8edbe
I20260926 07:25:20.288442 5912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:20.294512 5912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5: Bootstrap starting.
I20260926 07:25:20.295485 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:20.287788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55850 (local address 127.0.98.62:42291)
0926 07:25:20.288453 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:20.288468 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:20.288532 (+ 64us) server_negotiation.cc:407] Connection header received
0926 07:25:20.289165 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:20.289181 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:20.289556 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:20.289897 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:20.290299 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:20.291307 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:20.292180 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:20.292647 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:20.292824 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:20.293756 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:20.293761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:20.293772 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:20.294145 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:20.294197 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:20.294216 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:20.294322 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:20.294484 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 07:25:20.294748 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":159,"threads_started":1}
I20260926 07:25:20.297008 5912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:20.297860 5912 log.cc:824] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:20.298136 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:25:20.300016 5912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5: No bootstrap required, opened a new log
I20260926 07:25:20.303632 5912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6612c449329b41a4a90c95c01888abd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42291 } }
I20260926 07:25:20.303869 5912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:20.303905 5912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6612c449329b41a4a90c95c01888abd5, State: Initialized, Role: FOLLOWER
I20260926 07:25:20.304281 5912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6612c449329b41a4a90c95c01888abd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42291 } }
I20260926 07:25:20.304396 5912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:20.304445 5912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:20.304519 5912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:20.305450 5912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6612c449329b41a4a90c95c01888abd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42291 } }
I20260926 07:25:20.305819 5912 leader_election.cc:302] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6612c449329b41a4a90c95c01888abd5; no voters:
I20260926 07:25:20.306341 5912 leader_election.cc:288] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:20.306427 5917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:20.306696 5917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [term 1 LEADER]: Becoming Leader. State: Replica: 6612c449329b41a4a90c95c01888abd5, State: Running, Role: LEADER
I20260926 07:25:20.306993 5917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6612c449329b41a4a90c95c01888abd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42291 } }
I20260926 07:25:20.307639 5912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:20.308696 5919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6612c449329b41a4a90c95c01888abd5. Latest consensus state: current_term: 1 leader_uuid: "6612c449329b41a4a90c95c01888abd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6612c449329b41a4a90c95c01888abd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42291 } } }
I20260926 07:25:20.308830 5919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:20.309093 5918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6612c449329b41a4a90c95c01888abd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6612c449329b41a4a90c95c01888abd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42291 } } }
I20260926 07:25:20.309181 5918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:20.309321 5926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:20.312345 5926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:20.317595 5926 catalog_manager.cc:1347] Generated new cluster ID: a7806cc1bb024bddb31fdc0f2ed4e2ff
I20260926 07:25:20.317670 5926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:20.329284 5926 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:20.330174 5926 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:20.337551 5926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6612c449329b41a4a90c95c01888abd5: Generated new TSK 0
I20260926 07:25:20.338188 5926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:25:20.407416 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:20.346386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.0.98.62:42291)
0926 07:25:20.346514 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:20.346519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:20.346659 (+ 140us) server_negotiation.cc:407] Connection header received
0926 07:25:20.346780 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:20.346783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:20.346842 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:20.346951 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:25:20.347201 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:20.348005 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:20.348868 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:20.349227 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:20.349344 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:20.349583 (+ 239us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:25:20.406902 (+ 57319us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 07:25:20.407274 (+ 372us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55864: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 07:25:20.407506 5915 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55864: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 07:25:20.407588 5941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:20.346278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42291 (local address 127.0.0.1:55864)
0926 07:25:20.346576 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:20.346587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:20.346680 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:20.346961 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:20.346963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:20.346969 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:25:20.347092 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:20.347104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:20.348142 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:20.348144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:20.348759 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:20.348766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:20.349331 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:20.349347 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:20.349410 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:20.349429 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:25:20.407212 (+ 57783us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 07:25:20.407398 (+ 186us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42291: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":99,"threads_started":1}
I20260926 07:25:20.408239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 5845
2026-09-26T07:25:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (324 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:25:20 dist-test-slave-g04g krb5kdc[5953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:25:20 dist-test-slave-g04g krb5kdc[5953](info): set up 2 sockets
Sep 26 07:25:20 dist-test-slave-g04g krb5kdc[5953](info): commencing operation
krb5kdc: starting...
W20260926 07:25:21.691447 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:25:21 dist-test-slave-g04g krb5kdc[5953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:25:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:25:21.810392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35373
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:21.917750 5966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:21.918036 5966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:21.918120 5966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:21.922245 5966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:21.922327 5966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:21.922366 5966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:21.922402 5966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:21.922441 5966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:21.927525 5966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:21.928532 5966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:21.929493 5966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:21.937588 5974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:21.937631 5966 server_base.cc:1034] running on GCE node
W20260926 07:25:21.937548 5972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:21.937572 5971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:21.938392 5966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:21.939527 5966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:21.940686 5966 hybrid_clock.cc:648] HybridClock initialized: now 1790407521940664 us; error 44 us; skew 500 ppm
Sep 26 07:25:21 dist-test-slave-g04g krb5kdc[5953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:21.943387 5966 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:25:21.944221 5966 webserver.cc:460] Webserver started at http://127.0.98.62:39675/ using document root <none> and password file <none>
I20260926 07:25:21.944777 5966 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:21.944864 5966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:21.945083 5966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:21.947109 5966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "71d230a267dc4dc1aae53feb11452c8a"
format_stamp: "Formatted at 2026-09-26 07:25:21 on dist-test-slave-g04g"
server_key: "32a509b457d0a23fc95ac50a1fbcc9b2"
server_key_iv: "3a3b43779f68ea85dca84345b918daad"
server_key_version: "encryptionkey@0"
I20260926 07:25:21.947654 5966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "71d230a267dc4dc1aae53feb11452c8a"
format_stamp: "Formatted at 2026-09-26 07:25:21 on dist-test-slave-g04g"
server_key: "32a509b457d0a23fc95ac50a1fbcc9b2"
server_key_iv: "3a3b43779f68ea85dca84345b918daad"
server_key_version: "encryptionkey@0"
I20260926 07:25:21.951186 5966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:21.953508 5981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:21.954751 5966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:21.954891 5966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "71d230a267dc4dc1aae53feb11452c8a"
format_stamp: "Formatted at 2026-09-26 07:25:21 on dist-test-slave-g04g"
server_key: "32a509b457d0a23fc95ac50a1fbcc9b2"
server_key_iv: "3a3b43779f68ea85dca84345b918daad"
server_key_version: "encryptionkey@0"
I20260926 07:25:21.955000 5966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:21.970592 5966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:21.974009 5966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:21.974210 5966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:21.981387 5966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35373
I20260926 07:25:21.981726 6033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35373 every 8 connection(s)
I20260926 07:25:21.982370 5966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:21.985345 6034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:21.985961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 5966
I20260926 07:25:21.986155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:21.986462 392 external_mini_cluster.cc:1428] Setting key 188f239e7dfa8815e370ef203596e398
I20260926 07:25:21.991562 6034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a: Bootstrap starting.
I20260926 07:25:21.994148 6034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:21.995014 6034 log.cc:824] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:25:21 dist-test-slave-g04g krb5kdc[5953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:21.996894 6034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a: No bootstrap required, opened a new log
I20260926 07:25:22.000207 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:21.987800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.0.98.62:35373)
0926 07:25:21.988164 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:21.988189 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:21.988223 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:25:21.988844 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:21.988860 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:21.989182 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:21.989453 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:21.990422 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:21.991476 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:21.992562 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:21.993003 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:21.993123 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:21.995864 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:21.995869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:21.995881 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:21.995907 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:21.997630 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:21.998065 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:21.998071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:21.998076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:21.998187 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:21.998457 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:21.998461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:21.998463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:21.998827 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:21.999015 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:21.999266 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 07:25:21.999536 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":123,"threads_started":1}
I20260926 07:25:22.000327 6034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d230a267dc4dc1aae53feb11452c8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35373 } }
I20260926 07:25:22.000587 6034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:22.000674 6034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d230a267dc4dc1aae53feb11452c8a, State: Initialized, Role: FOLLOWER
I20260926 07:25:22.001133 6034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d230a267dc4dc1aae53feb11452c8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35373 } }
I20260926 07:25:22.001276 6034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:22.001344 6034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:22.001436 6034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:22.002318 6034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d230a267dc4dc1aae53feb11452c8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35373 } }
I20260926 07:25:22.002636 6034 leader_election.cc:302] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71d230a267dc4dc1aae53feb11452c8a; no voters:
I20260926 07:25:22.002912 6034 leader_election.cc:288] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:22.003015 6039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:22.003286 6039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [term 1 LEADER]: Becoming Leader. State: Replica: 71d230a267dc4dc1aae53feb11452c8a, State: Running, Role: LEADER
I20260926 07:25:22.003556 6039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d230a267dc4dc1aae53feb11452c8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35373 } }
I20260926 07:25:22.004098 6034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:22.005044 6040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71d230a267dc4dc1aae53feb11452c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d230a267dc4dc1aae53feb11452c8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35373 } } }
I20260926 07:25:22.005062 6041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71d230a267dc4dc1aae53feb11452c8a. Latest consensus state: current_term: 1 leader_uuid: "71d230a267dc4dc1aae53feb11452c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71d230a267dc4dc1aae53feb11452c8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35373 } } }
I20260926 07:25:22.005190 6040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:22.005190 6041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:22.005585 6048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:22.008538 6048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:22.014145 6048 catalog_manager.cc:1347] Generated new cluster ID: 1047d0c9854b4961b1397e7b6f5759d9
I20260926 07:25:22.014218 6048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:22.029524 6048 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:22.030493 6048 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:22.035281 6048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71d230a267dc4dc1aae53feb11452c8a: Generated new TSK 0
I20260926 07:25:22.035948 6048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:25:22.070883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35373
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:22.181491 6060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:22.181739 6060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:22.181782 6060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:22.185783 6060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:22.185837 6060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:22.185894 6060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:22.191064 6060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:22.192019 6060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:22.192955 6060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:22.203095 6065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:22.203126 6068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:22.203083 6066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:22.203303 6060 server_base.cc:1034] running on GCE node
I20260926 07:25:22.203879 6060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:22.204481 6060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:22.205655 6060 hybrid_clock.cc:648] HybridClock initialized: now 1790407522205630 us; error 47 us; skew 500 ppm
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:22.208448 6060 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:25:22.209287 6060 webserver.cc:460] Webserver started at http://127.0.98.1:39943/ using document root <none> and password file <none>
I20260926 07:25:22.209815 6060 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:22.209898 6060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:22.210119 6060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:22.212067 6060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "060b5af5543149b4af6b77123bac86fd"
format_stamp: "Formatted at 2026-09-26 07:25:22 on dist-test-slave-g04g"
server_key: "f5482195ad9d320928f3147b81f3c8ca"
server_key_iv: "7002223248f1ff3c10452c30c1cb853e"
server_key_version: "encryptionkey@0"
I20260926 07:25:22.212613 6060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "060b5af5543149b4af6b77123bac86fd"
format_stamp: "Formatted at 2026-09-26 07:25:22 on dist-test-slave-g04g"
server_key: "f5482195ad9d320928f3147b81f3c8ca"
server_key_iv: "7002223248f1ff3c10452c30c1cb853e"
server_key_version: "encryptionkey@0"
I20260926 07:25:22.215956 6060 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:22.218245 6075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:22.219571 6060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:22.219692 6060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "060b5af5543149b4af6b77123bac86fd"
format_stamp: "Formatted at 2026-09-26 07:25:22 on dist-test-slave-g04g"
server_key: "f5482195ad9d320928f3147b81f3c8ca"
server_key_iv: "7002223248f1ff3c10452c30c1cb853e"
server_key_version: "encryptionkey@0"
I20260926 07:25:22.219796 6060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:22.236121 6060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:22.239246 6060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:22.239431 6060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:22.239966 6060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:22.240832 6060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:22.240900 6060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:22.240967 6060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:22.241016 6060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:22.250283 6060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39859
I20260926 07:25:22.250423 6188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39859 every 8 connection(s)
I20260926 07:25:22.251431 6060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:22.256857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 6060
I20260926 07:25:22.256982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:25:22.257295 392 external_mini_cluster.cc:1428] Setting key df620bbf87b7182302d93e51abd9e2e0
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 07:25:22.267256 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:22.253501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41909 (local address 127.0.98.62:35373)
0926 07:25:22.253626 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:22.253631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:22.254635 (+ 1004us) server_negotiation.cc:407] Connection header received
0926 07:25:22.256362 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:22.256366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:22.256422 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:22.256525 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:22.257986 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.258724 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.260094 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.260444 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.260539 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:22.263481 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:22.263484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:22.263487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:22.263509 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:22.265022 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:22.265626 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:22.265629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:22.265630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:22.265709 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:22.266107 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:22.266109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:22.266110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:22.266269 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:22.266338 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:22.266966 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 07:25:22.267137 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:25:22.268131 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:22.254048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35373 (local address 127.0.98.1:41909)
0926 07:25:22.254489 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:22.254539 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:22.256099 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:22.256681 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:22.256691 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:22.257188 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:22.257824 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:22.257839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.258867 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.258870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:22.259974 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:22.259983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.260599 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.260606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:22.260821 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:22.261409 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:22.261427 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:22.263317 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:22.265140 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:22.265148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:22.265164 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:22.265519 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:22.265803 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:22.265809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:22.265811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:22.266007 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:22.266377 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:22.266387 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:22.266693 (+ 306us) client_negotiation.cc:769] Sending connection context
0926 07:25:22.266892 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 07:25:22.267267 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":127,"threads_started":1}
I20260926 07:25:22.269299 6189 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35373
I20260926 07:25:22.269517 6189 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 07:25:22.270009 6189 heartbeater.cc:511] Master 127.0.98.62:35373 requested a full tablet report, sending...
I20260926 07:25:22.271607 5996 ts_manager.cc:194] Registered new tserver with Master: 060b5af5543149b4af6b77123bac86fd (127.0.98.1:39859)
I20260926 07:25:22.272827 5996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:25:22.284181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35373
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:22.393712 6194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:22.393965 6194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:22.394006 6194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:22.397927 6194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:22.397982 6194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:22.398038 6194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:22.403355 6194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:22.404318 6194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:22.405428 6194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:22.415017 6199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:22.415011 6200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:22.415011 6202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:22.415321 6194 server_base.cc:1034] running on GCE node
I20260926 07:25:22.415814 6194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:22.416450 6194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:22.417635 6194 hybrid_clock.cc:648] HybridClock initialized: now 1790407522417614 us; error 34 us; skew 500 ppm
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:22.420491 6194 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:25:22.421350 6194 webserver.cc:460] Webserver started at http://127.0.98.2:46425/ using document root <none> and password file <none>
I20260926 07:25:22.421844 6194 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:22.421901 6194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:22.422058 6194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:22.424213 6194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "4f1dab10287c4189ae0ba6f89970a103"
format_stamp: "Formatted at 2026-09-26 07:25:22 on dist-test-slave-g04g"
server_key: "c631934d78d05a6c6b86f0cd058f821b"
server_key_iv: "9118a8125774cd3e27707198e3deca48"
server_key_version: "encryptionkey@0"
I20260926 07:25:22.424707 6194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4f1dab10287c4189ae0ba6f89970a103"
format_stamp: "Formatted at 2026-09-26 07:25:22 on dist-test-slave-g04g"
server_key: "c631934d78d05a6c6b86f0cd058f821b"
server_key_iv: "9118a8125774cd3e27707198e3deca48"
server_key_version: "encryptionkey@0"
I20260926 07:25:22.428246 6194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:22.430471 6209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:22.431697 6194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 07:25:22.431803 6194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "4f1dab10287c4189ae0ba6f89970a103"
format_stamp: "Formatted at 2026-09-26 07:25:22 on dist-test-slave-g04g"
server_key: "c631934d78d05a6c6b86f0cd058f821b"
server_key_iv: "9118a8125774cd3e27707198e3deca48"
server_key_version: "encryptionkey@0"
I20260926 07:25:22.431885 6194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:22.445989 6194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:22.448740 6194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:22.448884 6194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:22.449378 6194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:22.450103 6194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:22.450157 6194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:22.450196 6194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:22.450217 6194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:22.458623 6194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40957
I20260926 07:25:22.458691 6322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40957 every 8 connection(s)
I20260926 07:25:22.459832 6194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:22.469990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 6194
I20260926 07:25:22.470131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:25:22.470409 392 external_mini_cluster.cc:1428] Setting key ec1bb96752fa704641acdae72fa5a831
I20260926 07:25:22.473361 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:22.461594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42487 (local address 127.0.98.62:35373)
0926 07:25:22.461751 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:22.461756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:22.462590 (+ 834us) server_negotiation.cc:407] Connection header received
0926 07:25:22.463964 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:22.463968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:22.464015 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:22.464089 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:22.465194 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.465704 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.466774 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.467007 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.467201 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:22.469601 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:22.469604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:22.469606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:22.469630 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:22.471353 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:22.471898 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:22.471904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:22.471906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:22.471987 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:22.472324 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:22.472329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:22.472331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:22.472506 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:22.472618 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:22.473090 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 07:25:22.473205 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:25:22.474185 6325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:22.462009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35373 (local address 127.0.98.2:42487)
0926 07:25:22.462439 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:22.462481 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:22.463769 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:22.464252 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:22.464258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:22.464592 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:22.465069 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:22.465082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.465808 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.465810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:22.466671 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:22.466679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.467252 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.467255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:22.467399 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:22.468026 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:22.468041 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:22.469452 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:22.471458 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:22.471463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:22.471480 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:22.471793 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:22.472085 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:22.472089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:22.472091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:22.472229 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:22.472608 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:22.472615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:22.472853 (+ 238us) client_negotiation.cc:769] Sending connection context
0926 07:25:22.473040 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 07:25:22.473364 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260926 07:25:22.475405 6323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35373
I20260926 07:25:22.475653 6323 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:22.476187 6323 heartbeater.cc:511] Master 127.0.98.62:35373 requested a full tablet report, sending...
I20260926 07:25:22.477197 5996 ts_manager.cc:194] Registered new tserver with Master: 4f1dab10287c4189ae0ba6f89970a103 (127.0.98.2:40957)
I20260926 07:25:22.477764 5996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42487
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:25:22.498279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35373
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:22.606415 6328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:22.606645 6328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:22.606686 6328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:22.610770 6328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:22.610821 6328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:22.610903 6328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:22.616075 6328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:22.617018 6328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:22.617973 6328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:22.630141 6334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:22.630112 6333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:22.630132 6336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:22.630507 6328 server_base.cc:1034] running on GCE node
I20260926 07:25:22.630900 6328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:22.631579 6328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:22.632766 6328 hybrid_clock.cc:648] HybridClock initialized: now 1790407522632738 us; error 41 us; skew 500 ppm
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:22.635553 6328 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:25:22.636440 6328 webserver.cc:460] Webserver started at http://127.0.98.3:35993/ using document root <none> and password file <none>
I20260926 07:25:22.637044 6328 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:22.637130 6328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:22.637337 6328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:22.639369 6328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "a07949ecd4534fe884e9da17d66e30e7"
format_stamp: "Formatted at 2026-09-26 07:25:22 on dist-test-slave-g04g"
server_key: "895e3b90dbc7b647de2935941cb61273"
server_key_iv: "4c4c16b84c7bbb8f7fe84542150e9adb"
server_key_version: "encryptionkey@0"
I20260926 07:25:22.639923 6328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a07949ecd4534fe884e9da17d66e30e7"
format_stamp: "Formatted at 2026-09-26 07:25:22 on dist-test-slave-g04g"
server_key: "895e3b90dbc7b647de2935941cb61273"
server_key_iv: "4c4c16b84c7bbb8f7fe84542150e9adb"
server_key_version: "encryptionkey@0"
I20260926 07:25:22.643543 6328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:22.646049 6343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:22.647439 6328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:22.647572 6328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "a07949ecd4534fe884e9da17d66e30e7"
format_stamp: "Formatted at 2026-09-26 07:25:22 on dist-test-slave-g04g"
server_key: "895e3b90dbc7b647de2935941cb61273"
server_key_iv: "4c4c16b84c7bbb8f7fe84542150e9adb"
server_key_version: "encryptionkey@0"
I20260926 07:25:22.647706 6328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:22.664407 6328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:22.667601 6328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:22.667814 6328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:22.668362 6328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:22.669124 6328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:22.669193 6328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:22.669261 6328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:22.669303 6328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:22.677906 6328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35057
I20260926 07:25:22.677929 6456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35057 every 8 connection(s)
I20260926 07:25:22.678750 6328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:22.684222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 6328
I20260926 07:25:22.684305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:22.684576 392 external_mini_cluster.cc:1428] Setting key a37411baf1ed9c6df4031fbe369c3859
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:22.691993 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:22.680474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33661 (local address 127.0.98.62:35373)
0926 07:25:22.680600 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:22.680604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:22.681370 (+ 766us) server_negotiation.cc:407] Connection header received
0926 07:25:22.682652 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:22.682655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:22.682701 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:22.682774 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:22.683920 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.684485 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.686081 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.686432 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.686540 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:22.688898 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:22.688901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:22.688903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:22.688924 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:22.690310 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:22.690723 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:22.690725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:22.690727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:22.690796 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:22.691119 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:22.691121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:22.691122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:22.691288 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:22.691362 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:22.691795 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 07:25:22.691882 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:25:22.692751 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:22.680826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35373 (local address 127.0.98.3:33661)
0926 07:25:22.681227 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:22.681267 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:22.682465 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:22.682926 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:22.682932 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:22.683345 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:22.683795 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:22.683808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.684614 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.684618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:22.685966 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:22.685975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.686607 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.686612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:22.686766 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:22.687338 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:22.687354 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:22.688758 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:22.690400 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:22.690403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:22.690414 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:22.690645 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:22.690872 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:22.690874 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:22.690876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:22.690993 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:22.691372 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:22.691377 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:22.691582 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 07:25:22.691736 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 07:25:22.691960 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":103,"threads_started":1}
I20260926 07:25:22.693801 6457 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35373
I20260926 07:25:22.694024 6457 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:22.694482 6457 heartbeater.cc:511] Master 127.0.98.62:35373 requested a full tablet report, sending...
I20260926 07:25:22.695498 5996 ts_manager.cc:194] Registered new tserver with Master: a07949ecd4534fe884e9da17d66e30e7 (127.0.98.3:35057)
I20260926 07:25:22.696019 5996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33661
I20260926 07:25:22.698189 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/test-admin.keytab.
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:22.713346 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 07:25:22.716243 6464 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:22.714697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.0.98.1:39859)
0926 07:25:22.715096 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:22.715108 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:22.715140 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:25:22.715189 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:22.715194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:22.715383 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:22.715505 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:22.716063 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47422: BlockingRecv error: recv got EOF from 127.0.0.1:47422 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":113,"threads_started":1}
W20260926 07:25:22.716238 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:22.717347 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:22.718489 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 07:25:22.728300 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:22.719507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.0.98.1:39859)
0926 07:25:22.719644 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:22.719650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:22.719729 (+ 79us) server_negotiation.cc:407] Connection header received
0926 07:25:22.719880 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:22.719883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:22.719930 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:22.720014 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:22.720579 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.721347 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.722172 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.722569 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.722686 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:22.725013 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:22.725022 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:22.725045 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:22.725067 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:22.726500 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:22.726914 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:22.726921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:22.726923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:22.727005 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:22.727300 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:22.727304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:22.727306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:22.727541 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:22.727774 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:22.727937 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 07:25:22.728084 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 07:25:22.731935 6464 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:22.730973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.0.98.1:39859)
0926 07:25:22.731114 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:22.731121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:22.731275 (+ 154us) server_negotiation.cc:407] Connection header received
0926 07:25:22.731347 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:22.731349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:22.731381 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:22.731441 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:22.731809 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47444: BlockingRecv error: recv got EOF from 127.0.0.1:47444 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 07:25:22.732007 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:22.733003 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:22.733835 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 07:25:22.742123 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:22.734577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.0.98.1:39859)
0926 07:25:22.734690 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:22.734693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:22.734763 (+ 70us) server_negotiation.cc:407] Connection header received
0926 07:25:22.734875 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:22.734878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:22.734919 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:22.735013 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:22.735493 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.736094 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.736810 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.737022 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.737136 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:22.739264 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:22.739267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:22.739269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:22.739295 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:22.740758 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:22.741085 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:22.741087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:22.741089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:22.741155 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:22.741395 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:22.741398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:22.741399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:22.741624 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:22.741747 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:22.741899 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 07:25:22.742001 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 07:25:22.744867 6464 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:22.743801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.0.98.1:39859)
0926 07:25:22.743921 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:22.743926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:22.744053 (+ 127us) server_negotiation.cc:407] Connection header received
0926 07:25:22.744211 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:22.744215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:22.744259 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:22.744347 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:22.744754 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47454: BlockingRecv error: recv got EOF from 127.0.0.1:47454 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 07:25:22.745003 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790407489657243-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:22.745927 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:22.746783 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[5953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 07:25:22.755242 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:22.747537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.0.98.1:39859)
0926 07:25:22.747661 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:22.747665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:22.747875 (+ 210us) server_negotiation.cc:407] Connection header received
0926 07:25:22.747911 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:22.747914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:22.747957 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:22.748067 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:22.748477 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.748947 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.749675 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:22.749877 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:22.749951 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:22.752265 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:22.752268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:22.752270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:22.752291 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:22.753856 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:22.754201 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:22.754203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:22.754204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:22.754279 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:22.754494 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:22.754496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:22.754497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:22.754655 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:22.754756 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:22.754951 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 07:25:22.755092 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:25:22.756201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 6060
I20260926 07:25:22.762079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 6194
I20260926 07:25:22.768559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 6328
I20260926 07:25:22.774128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 5966
2026-09-26T07:25:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2121 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[6477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[6477](info): set up 2 sockets
Sep 26 07:25:22 dist-test-slave-g04g krb5kdc[6477](info): commencing operation
krb5kdc: starting...
W20260926 07:25:23.843472 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:25:23 dist-test-slave-g04g krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:25:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 07:25:23.967265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33023
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:24.080986 6490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:24.081254 6490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:24.081326 6490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:24.085465 6490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:24.085527 6490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:24.085590 6490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:24.085619 6490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:24.085656 6490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:24.090819 6490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:24.091871 6490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:24.092904 6490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:24.101984 6496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:24.102003 6495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:24.102149 6490 server_base.cc:1034] running on GCE node
W20260926 07:25:24.101984 6498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:24.102846 6490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:24.103971 6490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:24.105165 6490 hybrid_clock.cc:648] HybridClock initialized: now 1790407524105150 us; error 40 us; skew 500 ppm
Sep 26 07:25:24 dist-test-slave-g04g krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:24.108072 6490 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 07:25:24.108982 6490 webserver.cc:460] Webserver started at http://127.0.98.62:33075/ using document root <none> and password file <none>
I20260926 07:25:24.109529 6490 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:24.109615 6490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:24.109839 6490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:24.111872 6490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "46dc84ef76be4b5181e29f5cd68aacd5"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "9f77e21b329209db90e6d0b0046cca15"
server_key_iv: "71bc080bc327f17b96c44cb21318c988"
server_key_version: "encryptionkey@0"
I20260926 07:25:24.112417 6490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "46dc84ef76be4b5181e29f5cd68aacd5"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "9f77e21b329209db90e6d0b0046cca15"
server_key_iv: "71bc080bc327f17b96c44cb21318c988"
server_key_version: "encryptionkey@0"
I20260926 07:25:24.116083 6490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:24.118460 6505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:24.119704 6490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:24.119836 6490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "46dc84ef76be4b5181e29f5cd68aacd5"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "9f77e21b329209db90e6d0b0046cca15"
server_key_iv: "71bc080bc327f17b96c44cb21318c988"
server_key_version: "encryptionkey@0"
I20260926 07:25:24.119974 6490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:24.151427 6490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:24.154819 6490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:24.155067 6490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:24.162173 6490 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33023
I20260926 07:25:24.162467 6557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33023 every 8 connection(s)
I20260926 07:25:24.163187 6490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:24.163450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 6490
I20260926 07:25:24.163578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:24.163944 392 external_mini_cluster.cc:1428] Setting key b55dc83118b823f1baccfa9a2e46e03f
I20260926 07:25:24.166392 6558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:24.172706 6558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5: Bootstrap starting.
Sep 26 07:25:24 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:24.175338 6558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:24.176213 6558 log.cc:824] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:24.178726 6558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5: No bootstrap required, opened a new log
I20260926 07:25:24.179466 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:24.165313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.0.98.62:33023)
0926 07:25:24.165711 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:24.165729 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:24.165771 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:25:24.166622 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:24.166642 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:24.167076 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:24.167429 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:24.168197 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.169695 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.170599 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.171315 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.171472 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:24.174363 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:24.174369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:24.174383 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:24.174412 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:24.176540 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:24.176935 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:24.176939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:24.176943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:24.177047 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:24.177361 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:24.177364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:24.177366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:24.177820 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:24.177990 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:24.178231 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 07:25:24.178506 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
I20260926 07:25:24.182452 6558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } }
I20260926 07:25:24.182698 6558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:24.182785 6558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46dc84ef76be4b5181e29f5cd68aacd5, State: Initialized, Role: FOLLOWER
I20260926 07:25:24.183301 6558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } }
I20260926 07:25:24.183475 6558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:24.183539 6558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:24.183629 6558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:24.184448 6558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } }
I20260926 07:25:24.184770 6558 leader_election.cc:302] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46dc84ef76be4b5181e29f5cd68aacd5; no voters:
I20260926 07:25:24.185078 6558 leader_election.cc:288] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:24.185169 6563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:24.185410 6563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [term 1 LEADER]: Becoming Leader. State: Replica: 46dc84ef76be4b5181e29f5cd68aacd5, State: Running, Role: LEADER
I20260926 07:25:24.185731 6563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } }
I20260926 07:25:24.186184 6558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:24.187418 6565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46dc84ef76be4b5181e29f5cd68aacd5. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } }
I20260926 07:25:24.187563 6565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:24.187413 6564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } }
I20260926 07:25:24.187716 6564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:24.188381 6572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:24.191248 6572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:24.195961 6572 catalog_manager.cc:1347] Generated new cluster ID: 28dc622c9eb64edebd3dde2b34769ebe
I20260926 07:25:24.196043 6572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:24.205032 6572 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:24.206146 6572 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:24.210741 6572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5: Generated new TSK 0
I20260926 07:25:24.211350 6572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 07:25:24.239758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33023
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:24.359308 6584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:24.359560 6584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:24.359637 6584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:24.363628 6584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:24.363684 6584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:24.363790 6584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:24.368875 6584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:24.369859 6584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:24.370858 6584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:24.380203 6590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:24.380216 6592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:24.380203 6589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:24.380569 6584 server_base.cc:1034] running on GCE node
I20260926 07:25:24.380939 6584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:24.381521 6584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:24.382688 6584 hybrid_clock.cc:648] HybridClock initialized: now 1790407524382670 us; error 30 us; skew 500 ppm
Sep 26 07:25:24 dist-test-slave-g04g krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:24.385351 6584 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 07:25:24.386224 6584 webserver.cc:460] Webserver started at http://127.0.98.1:37607/ using document root <none> and password file <none>
I20260926 07:25:24.386839 6584 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:24.386907 6584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:24.387188 6584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:24.389138 6584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "852fb4b9b0fb4ed187c9f283d7bc3132"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "53f99c334c545bd58288ea2dad9bbe5e"
server_key_iv: "bcea670fc16ec03e39ff3edf187b69bf"
server_key_version: "encryptionkey@0"
I20260926 07:25:24.389693 6584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "852fb4b9b0fb4ed187c9f283d7bc3132"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "53f99c334c545bd58288ea2dad9bbe5e"
server_key_iv: "bcea670fc16ec03e39ff3edf187b69bf"
server_key_version: "encryptionkey@0"
I20260926 07:25:24.393321 6584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 07:25:24.395594 6599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:24.396767 6584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:24.396893 6584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "852fb4b9b0fb4ed187c9f283d7bc3132"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "53f99c334c545bd58288ea2dad9bbe5e"
server_key_iv: "bcea670fc16ec03e39ff3edf187b69bf"
server_key_version: "encryptionkey@0"
I20260926 07:25:24.396997 6584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:24.409475 6584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:24.412415 6584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:24.412604 6584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:24.413141 6584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:24.413908 6584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:24.413976 6584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:24.414047 6584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:24.414088 6584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:24.422427 6584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37491
I20260926 07:25:24.422442 6712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37491 every 8 connection(s)
I20260926 07:25:24.423424 6584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:24.425767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 6584
I20260926 07:25:24.425885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:25:24.426230 392 external_mini_cluster.cc:1428] Setting key 79d3b619667e71ffa8a2c00787b19474
Sep 26 07:25:24 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 07:25:24.439164 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:24.425167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56525 (local address 127.0.98.62:33023)
0926 07:25:24.425328 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:24.425333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:24.426228 (+ 895us) server_negotiation.cc:407] Connection header received
0926 07:25:24.427893 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:24.427896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:24.427942 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:24.428012 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:24.429622 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.430496 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.432360 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.432608 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.432692 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:24.435543 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:24.435549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:24.435553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:24.435587 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:24.437085 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:24.437596 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:24.437606 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:24.437608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:24.437684 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:24.438025 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:24.438028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:24.438030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:24.438205 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:24.438313 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:24.438843 (+ 530us) server_negotiation.cc:299] Negotiation successful
0926 07:25:24.438967 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260926 07:25:24.440099 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:24.425550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.1:56525)
0926 07:25:24.426054 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:24.426115 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:24.427660 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:24.428217 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:24.428229 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:24.428751 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:24.429444 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:24.429466 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.430647 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.430653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:24.432239 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:24.432252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.432765 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.432773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:24.433018 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:24.433657 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:24.433685 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:24.435371 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:24.437185 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:24.437190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:24.437201 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:24.437481 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:24.437775 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:24.437780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:24.437782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:24.437922 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:24.438338 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:24.438344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:24.438604 (+ 260us) client_negotiation.cc:769] Sending connection context
0926 07:25:24.438789 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 07:25:24.439197 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":101,"threads_started":1}
I20260926 07:25:24.441313 6713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33023
I20260926 07:25:24.441573 6713 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:24.442080 6713 heartbeater.cc:511] Master 127.0.98.62:33023 requested a full tablet report, sending...
I20260926 07:25:24.443701 6522 ts_manager.cc:194] Registered new tserver with Master: 852fb4b9b0fb4ed187c9f283d7bc3132 (127.0.98.1:37491)
I20260926 07:25:24.445410 6522 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 07:25:24.456283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33023
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:24.563699 6718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:24.563953 6718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:24.563998 6718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:24.567947 6718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:24.568003 6718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:24.568061 6718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:24.573149 6718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:24.574103 6718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:24.575114 6718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:24.584380 6724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:24.584388 6723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:24.584625 6718 server_base.cc:1034] running on GCE node
W20260926 07:25:24.584347 6726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:24.585141 6718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:24.585807 6718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:24.586995 6718 hybrid_clock.cc:648] HybridClock initialized: now 1790407524586973 us; error 33 us; skew 500 ppm
Sep 26 07:25:24 dist-test-slave-g04g krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:24.590118 6718 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 07:25:24.590925 6718 webserver.cc:460] Webserver started at http://127.0.98.2:34931/ using document root <none> and password file <none>
I20260926 07:25:24.591553 6718 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:24.591614 6718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:24.591773 6718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:24.593752 6718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "ef7a5385aae84f3192e11aa9e82c233a"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "4e16f63ddcdb4d2fd5929aeb2312980d"
server_key_iv: "5eb82c47232dbe657d4d00badaa2ab70"
server_key_version: "encryptionkey@0"
I20260926 07:25:24.594261 6718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ef7a5385aae84f3192e11aa9e82c233a"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "4e16f63ddcdb4d2fd5929aeb2312980d"
server_key_iv: "5eb82c47232dbe657d4d00badaa2ab70"
server_key_version: "encryptionkey@0"
I20260926 07:25:24.598012 6718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:25:24.600492 6733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:24.601743 6718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:24.601838 6718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "ef7a5385aae84f3192e11aa9e82c233a"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "4e16f63ddcdb4d2fd5929aeb2312980d"
server_key_iv: "5eb82c47232dbe657d4d00badaa2ab70"
server_key_version: "encryptionkey@0"
I20260926 07:25:24.601926 6718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:24.615017 6718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:24.617900 6718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:24.618106 6718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:24.618714 6718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:24.619572 6718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:24.619643 6718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:24.619711 6718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:24.619769 6718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:24.631904 6718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36701
I20260926 07:25:24.631924 6846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36701 every 8 connection(s)
I20260926 07:25:24.632825 6718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:25:24 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:24.642249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 6718
I20260926 07:25:24.642376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:25:24.642706 392 external_mini_cluster.cc:1428] Setting key 643cdc17f6f16705ffb8b0c10938b227
I20260926 07:25:24.646678 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:24.634575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47989 (local address 127.0.98.62:33023)
0926 07:25:24.634741 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:24.634745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:24.635630 (+ 885us) server_negotiation.cc:407] Connection header received
0926 07:25:24.636897 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:24.636900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:24.636945 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:24.637025 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:24.638146 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.638659 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.639802 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.640015 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.640097 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:24.642594 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:24.642597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:24.642599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:24.642621 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:24.644412 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:24.645022 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:24.645025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:24.645027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:24.645099 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:24.645502 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:24.645505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:24.645506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:24.645739 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:24.645879 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:24.646357 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 07:25:24.646495 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:25:24.647410 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:24.634996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.2:47989)
0926 07:25:24.635483 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:24.635526 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:24.636693 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:24.637179 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:24.637186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:24.637529 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:24.638009 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:24.638021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.638761 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.638763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:24.639682 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:24.639691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.640140 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.640143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:24.640273 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:24.640863 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:24.640878 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:24.642376 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:24.644523 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:24.644531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:24.644547 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:24.644899 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:24.645190 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:24.645196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:24.645201 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:24.645381 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:24.645886 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:24.645895 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:24.646124 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 07:25:24.646275 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 07:25:24.646505 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":109,"threads_started":1}
I20260926 07:25:24.648552 6847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33023
I20260926 07:25:24.648815 6847 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:24.649410 6847 heartbeater.cc:511] Master 127.0.98.62:33023 requested a full tablet report, sending...
I20260926 07:25:24.650508 6522 ts_manager.cc:194] Registered new tserver with Master: ef7a5385aae84f3192e11aa9e82c233a (127.0.98.2:36701)
I20260926 07:25:24.651108 6522 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47989
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 07:25:24.670207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33023
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:24.780092 6852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:24.780356 6852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:24.780452 6852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:24.784493 6852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:24.784585 6852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:24.784674 6852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:24.790073 6852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:24.791363 6852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:24.792452 6852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:24.802353 6860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:24.802353 6857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:24.802448 6852 server_base.cc:1034] running on GCE node
W20260926 07:25:24.802350 6858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:24.802983 6852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:24.803751 6852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:24.804915 6852 hybrid_clock.cc:648] HybridClock initialized: now 1790407524804900 us; error 32 us; skew 500 ppm
Sep 26 07:25:24 dist-test-slave-g04g krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:24.808019 6852 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 07:25:24.808881 6852 webserver.cc:460] Webserver started at http://127.0.98.3:36135/ using document root <none> and password file <none>
I20260926 07:25:24.809442 6852 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:24.809502 6852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:24.809711 6852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:24.811782 6852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "c08cb6a0209647a49169edae08439ffc"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "797038a272ff2b29e7a46a1f1158e0a3"
server_key_iv: "9e54d8a8865d4f01cc0d231013d13aaa"
server_key_version: "encryptionkey@0"
I20260926 07:25:24.812330 6852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c08cb6a0209647a49169edae08439ffc"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "797038a272ff2b29e7a46a1f1158e0a3"
server_key_iv: "9e54d8a8865d4f01cc0d231013d13aaa"
server_key_version: "encryptionkey@0"
I20260926 07:25:24.815960 6852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:24.818259 6867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:24.819555 6852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:24.819685 6852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "c08cb6a0209647a49169edae08439ffc"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "797038a272ff2b29e7a46a1f1158e0a3"
server_key_iv: "9e54d8a8865d4f01cc0d231013d13aaa"
server_key_version: "encryptionkey@0"
I20260926 07:25:24.819796 6852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:24.835911 6852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:24.838820 6852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:24.839066 6852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:24.839648 6852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:24.840438 6852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:24.840509 6852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:24.840579 6852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:24.840621 6852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:24.849640 6852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34843
I20260926 07:25:24.849676 6980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34843 every 8 connection(s)
I20260926 07:25:24.850564 6852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:24.856244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 6852
I20260926 07:25:24.856330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:24.856602 392 external_mini_cluster.cc:1428] Setting key 535a128858d50103cd8e40353b72ca89
Sep 26 07:25:24 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:24.863942 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:24.852448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51871 (local address 127.0.98.62:33023)
0926 07:25:24.852608 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:24.852612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:24.853362 (+ 750us) server_negotiation.cc:407] Connection header received
0926 07:25:24.854701 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:24.854704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:24.854763 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:24.854869 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:24.856099 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.856753 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.857879 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.858103 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.858213 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:24.860571 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:24.860575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:24.860577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:24.860600 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:24.862068 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:24.862555 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:24.862557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:24.862559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:24.862627 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:24.862944 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:24.862947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:24.862948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:24.863164 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:24.863243 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:24.863723 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 07:25:24.863815 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 07:25:24.864749 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:24.852783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.3:51871)
0926 07:25:24.853206 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:24.853247 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:24.854482 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:24.855075 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:24.855086 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:24.855449 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:24.855915 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:24.855929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.856877 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.856881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:24.857750 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:24.857758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:24.858246 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:24.858250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:24.858419 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:24.858913 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:24.858928 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:24.860419 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:24.862180 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:24.862184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:24.862196 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:24.862469 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:24.862707 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:24.862709 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:24.862710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:24.862856 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:24.863280 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:24.863285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:24.863522 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 07:25:24.863683 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 07:25:24.863930 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":104,"threads_started":1}
I20260926 07:25:24.865844 6981 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33023
I20260926 07:25:24.866075 6981 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:24.866534 6981 heartbeater.cc:511] Master 127.0.98.62:33023 requested a full tablet report, sending...
I20260926 07:25:24.867547 6522 ts_manager.cc:194] Registered new tserver with Master: c08cb6a0209647a49169edae08439ffc (127.0.98.3:34843)
I20260926 07:25:24.868106 6522 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51871
I20260926 07:25:24.869995 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260926 07:25:24.895848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38947
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33023,127.0.98.61:38947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:25.006480 6986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:25.006819 6986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:25.006906 6986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:25.010993 6986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:25.011101 6986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:25.011142 6986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:25.011179 6986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:25.011211 6986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:25.016435 6986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33023,127.0.98.61:38947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
W20260926 07:25:25.016703 6986 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:33023,127.0.98.61:38947'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 07:25:25.017617 6986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:25.018656 6986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:25.027275 6992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:25.027282 6991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:25.027316 6986 server_base.cc:1034] running on GCE node
W20260926 07:25:25.027293 6994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:25.028019 6986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:25.029011 6986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:25.030165 6986 hybrid_clock.cc:648] HybridClock initialized: now 1790407525030150 us; error 27 us; skew 500 ppm
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:25.032851 6986 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 07:25:25.033653 6986 webserver.cc:460] Webserver started at http://127.0.98.61:45711/ using document root <none> and password file <none>
I20260926 07:25:25.034176 6986 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:25.034235 6986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:25.034480 6986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:25.036450 6986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/data/instance:
uuid: "fd9da486a92542fcb2304179a2ab9f4c"
format_stamp: "Formatted at 2026-09-26 07:25:25 on dist-test-slave-g04g"
server_key: "84aa4a6d4f0e4cbd786d518914f8130c"
server_key_iv: "474b6f8200248bbc592dd7e58a103a8c"
server_key_version: "encryptionkey@0"
I20260926 07:25:25.036988 6986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/wal/instance:
uuid: "fd9da486a92542fcb2304179a2ab9f4c"
format_stamp: "Formatted at 2026-09-26 07:25:25 on dist-test-slave-g04g"
server_key: "84aa4a6d4f0e4cbd786d518914f8130c"
server_key_iv: "474b6f8200248bbc592dd7e58a103a8c"
server_key_version: "encryptionkey@0"
I20260926 07:25:25.040570 6986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:25:25.042819 7001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:25.044009 6986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:25.044140 6986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/wal
uuid: "fd9da486a92542fcb2304179a2ab9f4c"
format_stamp: "Formatted at 2026-09-26 07:25:25 on dist-test-slave-g04g"
server_key: "84aa4a6d4f0e4cbd786d518914f8130c"
server_key_iv: "474b6f8200248bbc592dd7e58a103a8c"
server_key_version: "encryptionkey@0"
I20260926 07:25:25.044240 6986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:25.068701 6986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:25.071630 6986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:25.071830 6986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:25.078619 6986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38947
I20260926 07:25:25.078919 7053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38947 every 8 connection(s)
I20260926 07:25:25.079609 6986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/data/info.pb
I20260926 07:25:25.081755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 6986
I20260926 07:25:25.082517 7054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:25.086386 7054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:25.096997 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.083241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59346 (local address 127.0.98.61:38947)
0926 07:25:25.083647 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.083662 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.083696 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:25:25.084327 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.084343 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.084638 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.084977 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.085729 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.086885 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.087950 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.088649 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.088835 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.091862 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.091866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.091875 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.091898 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.093831 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.094267 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.094275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.094283 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.094402 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.094802 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.094807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.094811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.095279 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.095495 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.095756 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.096043 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":134,"threads_started":1}
I20260926 07:25:25.097863 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.088232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.0.98.62:33023)
0926 07:25:25.088373 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.088378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.088680 (+ 302us) server_negotiation.cc:407] Connection header received
0926 07:25:25.088833 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.088837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.088883 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.088937 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.089644 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.090503 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.091700 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.091986 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.092095 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.094368 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.094371 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.094373 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.094394 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.096178 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.096633 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.096635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.096636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.096700 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.096955 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.096957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.096958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.097242 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.097319 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.097641 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.097725 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:25:25.097924 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.088176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:49300)
0926 07:25:25.088549 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.088597 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.088714 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.089096 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.089099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.089402 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.089523 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.089527 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.090651 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.090654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.091587 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.091596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.092102 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.092105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.092220 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.092740 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.092755 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.094229 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.096271 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.096277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.096282 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.096541 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.096782 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.096785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.096787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.096872 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.097338 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.097343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.097458 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.097595 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.097745 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":139,"threads_started":1}
I20260926 07:25:25.099123 7054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 07:25:25.107764 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.099529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38947 (local address 127.0.0.1:59354)
0926 07:25:25.099635 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.099649 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.099720 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.099954 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.099957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.100105 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.100177 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.100181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.101235 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.101238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.101719 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.101725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.102293 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.102296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.102392 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.102800 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.102845 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.104380 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.106260 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.106263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.106265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.106513 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.106771 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.106776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.106778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.106862 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.107265 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.107268 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.107343 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.107466 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.107590 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 07:25:25.107772 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.099610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59354 (local address 127.0.98.61:38947)
0926 07:25:25.099769 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.099772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.099784 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:25:25.099812 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.099814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.099844 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.099945 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.100276 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.101130 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.101830 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.102204 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.102305 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.104506 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.104509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.104511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.104530 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.106136 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.106602 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.106608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.106611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.106685 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.106970 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.106973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.106975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.107175 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.107275 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.107508 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.107643 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":63}
I20260926 07:25:25.110718 7054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: Bootstrap starting.
I20260926 07:25:25.113106 7054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:25.113806 7054 log.cc:824] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:25.115728 7054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: No bootstrap required, opened a new log
I20260926 07:25:25.118544 7054 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } }
I20260926 07:25:25.118743 7054 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:25.118777 7054 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9da486a92542fcb2304179a2ab9f4c, State: Initialized, Role: FOLLOWER
I20260926 07:25:25.119284 7054 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } }
I20260926 07:25:25.119705 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } } }
I20260926 07:25:25.119836 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:25:25.120409 7054 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:25:25.124926 7071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:25:25.125011 7071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:25:25.126888 6986 master_runner.cc:428] Detected that this master fd9da486a92542fcb2304179a2ab9f4c is joining an existing cluster
I20260926 07:25:25.126948 6986 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:38947
I20260926 07:25:25.137032 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.128892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59368 (local address 127.0.98.61:38947)
0926 07:25:25.129236 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.129240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.129281 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:25:25.129410 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.129414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.129455 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.129529 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.129961 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.130636 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.131855 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.132165 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.132252 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.133574 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.133577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.133580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.133600 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.135330 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.135881 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.135886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.135888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.135963 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.136223 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.136226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.136227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.136385 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.136495 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.136740 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.136869 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":277,"spinlock_wait_cycles":2816}
I20260926 07:25:25.137187 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.128870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38947 (local address 127.0.0.1:59368)
0926 07:25:25.129211 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.129223 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.129321 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.129553 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.129555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.129765 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.129851 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.129857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.130747 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.130751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.131761 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.131772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.132365 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.132367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.132430 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.133064 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.133075 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.133439 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.135456 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.135460 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.135462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.135789 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.136049 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.136054 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.136056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.136130 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.136484 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.136486 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.136582 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.136680 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.136819 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":38,"spinlock_wait_cycles":4480,"thread_start_us":92,"threads_started":1}
I20260926 07:25:25.138042 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.129327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49314 (local address 127.0.98.62:33023)
0926 07:25:25.129439 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.129443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.129455 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:25:25.129496 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.129499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.129543 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.129621 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.130235 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.130921 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.131810 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.132146 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.132253 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.133686 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.133690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.133693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.133718 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.135291 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.135914 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.135919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.135920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.136004 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.136315 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.136319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.136321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.136519 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.136591 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.136931 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.137045 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:25:25.138700 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.128830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:49314)
0926 07:25:25.129103 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.129121 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.129210 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.129749 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.129752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.129964 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.130109 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.130117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.131021 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.131026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.131694 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.131702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.132273 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.132277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.132384 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.133008 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.133024 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.133532 (+ 508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.135427 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.135430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.135432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.135792 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.136118 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.136121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.136123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.136204 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.136665 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.136668 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.136752 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.138374 (+ 1622us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.138527 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":22,"spinlock_wait_cycles":3840,"thread_start_us":107,"threads_started":1}
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 07:25:25.152133 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.142674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38947 (local address 127.0.0.1:59374)
0926 07:25:25.143275 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.143313 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.143466 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.143767 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.143773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.144123 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.144217 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.144222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.144907 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.144912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.145552 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.145560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.145994 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.145997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.146107 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.146640 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.146652 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.148353 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.150215 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.150224 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.150230 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.150527 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.150839 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.150842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.150844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.150937 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.151465 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.151472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.151597 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.151752 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.151947 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":78,"threads_started":1}
I20260926 07:25:25.152173 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.142728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.0.98.61:38947)
0926 07:25:25.142838 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.142842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.143386 (+ 544us) server_negotiation.cc:407] Connection header received
0926 07:25:25.143599 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.143602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.143653 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.143735 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.144330 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.144802 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.145667 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.145878 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.145994 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.148499 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.148502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.148505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.148529 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.150092 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.150632 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.150638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.150640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.150728 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.151059 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.151065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.151070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.151350 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.151458 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.151823 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.151964 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:25:25.153853 6522 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:38947: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 38947
}
attrs {
promote: true
}
}
dest_uuid: "46dc84ef76be4b5181e29f5cd68aacd5"
cas_config_opid_index: -1
I20260926 07:25:25.154469 6522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true } }
I20260926 07:25:25.156085 7081 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fd9da486a92542fcb2304179a2ab9f4c (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true } } }
I20260926 07:25:25.156935 7028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:25.158344 7081 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:38947
I20260926 07:25:25.158329 7028 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 FOLLOWER]: Refusing update from remote peer 46dc84ef76be4b5181e29f5cd68aacd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 07:25:25.158855 6986 master.cc:537] Master@127.0.98.61:38947 shutting down...
I20260926 07:25:25.159420 7081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true } } }
I20260926 07:25:25.159432 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:25.159654 7081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: This master's current role is: LEADER
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
W20260926 07:25:25.161552 6506 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260926 07:25:25.164567 6986 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:25.164814 6986 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:25.164908 6986 tablet_replica.cc:330] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: stopping tablet replica
W20260926 07:25:25.165640 6506 consensus_peers.cc:487] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 -> Peer fd9da486a92542fcb2304179a2ab9f4c (127.0.98.61:38947): Couldn't send request to peer fd9da486a92542fcb2304179a2ab9f4c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260926 07:25:25.189494 6986 master.cc:552] Master@127.0.98.61:38947 shutdown complete.
I20260926 07:25:25.189555 6986 master_runner.cc:305] Clearing existing system tablet
I20260926 07:25:25.189790 6986 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:25.189844 6986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:25.190060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46361
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33023,127.0.98.61:38947,127.0.98.60:46361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:25:25.191165 6986 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:25:25.192080 7090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:25.192310 6986 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:25.192440 6986 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:25.192471 6986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/wal
uuid: "fd9da486a92542fcb2304179a2ab9f4c"
format_stamp: "Formatted at 2026-09-26 07:25:25 on dist-test-slave-g04g"
server_key: "84aa4a6d4f0e4cbd786d518914f8130c"
server_key_iv: "474b6f8200248bbc592dd7e58a103a8c"
server_key_version: "encryptionkey@0"
I20260926 07:25:25.194541 6986 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:25.196812 6986 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 07:25:25.196898 6986 log.cc:1196] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 07:25:25.198063 6986 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: Deleting consensus metadata
I20260926 07:25:25.198228 6986 master_runner.cc:315] Copying system tablet from 127.0.98.62:33023
I20260926 07:25:25.199398 6986 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33023
I20260926 07:25:25.208696 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.200392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.0.98.62:33023)
0926 07:25:25.200534 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.200541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.200560 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:25:25.200614 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.200620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.200678 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.200785 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.201382 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.202231 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.203434 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.203804 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.203905 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.205168 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.205172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.205175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.205200 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.206964 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.207452 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.207456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.207458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.207541 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.207822 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.207828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.207833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.208072 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.208171 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.208433 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.208542 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:25:25.208729 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.199976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:49330)
0926 07:25:25.200244 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.200261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.200370 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.200815 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.200818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.201097 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.201222 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.201229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.202357 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.202360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.203315 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.203325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.203926 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.203930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.204023 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.204591 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.204606 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.205032 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.207092 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.207096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.207098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.207339 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.207633 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.207636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.207638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.207723 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.208171 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.208174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.208266 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.208369 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.208527 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":94,"threads_started":1}
I20260926 07:25:25.209561 6542 tablet_copy_service.cc:140] P 46dc84ef76be4b5181e29f5cd68aacd5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fd9da486a92542fcb2304179a2ab9f4c ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49330)
I20260926 07:25:25.209734 6542 tablet_copy_service.cc:161] P 46dc84ef76be4b5181e29f5cd68aacd5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fd9da486a92542fcb2304179a2ab9f4c at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49330: session id = fd9da486a92542fcb2304179a2ab9f4c-00000000000000000000000000000000
I20260926 07:25:25.211113 6542 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 07:25:25.212659 6986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 07:25:25.214866 6986 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: tablet copy: Starting download of 0 data blocks...
I20260926 07:25:25.215076 6986 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: tablet copy: Starting download of 1 WAL segments...
I20260926 07:25:25.216749 6986 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 07:25:25.218468 6542 tablet_copy_service.cc:342] P 46dc84ef76be4b5181e29f5cd68aacd5: Request end of tablet copy session fd9da486a92542fcb2304179a2ab9f4c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49330
I20260926 07:25:25.218582 6542 tablet_copy_service.cc:433] P 46dc84ef76be4b5181e29f5cd68aacd5: ending tablet copy session fd9da486a92542fcb2304179a2ab9f4c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fd9da486a92542fcb2304179a2ab9f4c
I20260926 07:25:25.226025 6986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:25.227669 7097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:25.227739 7100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:25.227900 7098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:25.228088 6986 server_base.cc:1034] running on GCE node
I20260926 07:25:25.235275 6986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:25.235602 6986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:25.236754 6986 hybrid_clock.cc:648] HybridClock initialized: now 1790407525236734 us; error 44 us; skew 500 ppm
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:25.238365 6986 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 07:25:25.238786 6986 webserver.cc:460] Webserver started at http://127.0.98.61:36267/ using document root <none> and password file <none>
I20260926 07:25:25.238965 6986 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:25.239005 6986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:25.240466 6986 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:25.241430 7107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:25.241669 6986 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:25.241743 6986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/wal
uuid: "fd9da486a92542fcb2304179a2ab9f4c"
format_stamp: "Formatted at 2026-09-26 07:25:25 on dist-test-slave-g04g"
server_key: "84aa4a6d4f0e4cbd786d518914f8130c"
server_key_iv: "474b6f8200248bbc592dd7e58a103a8c"
server_key_version: "encryptionkey@0"
I20260926 07:25:25.241820 6986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:25.262192 6986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:25.264513 6986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:25.269744 7159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38947 every 8 connection(s)
I20260926 07:25:25.269764 6986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38947
I20260926 07:25:25.270560 6986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-1/data/info.pb
I20260926 07:25:25.272413 7160 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:25:25.273460 7160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: Bootstrap starting.
I20260926 07:25:25.283537 7160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:25:25.283839 7160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: Bootstrap complete.
I20260926 07:25:25.284468 7160 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true } }
I20260926 07:25:25.284600 7160 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fd9da486a92542fcb2304179a2ab9f4c, State: Initialized, Role: LEARNER
I20260926 07:25:25.284705 7160 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true } }
I20260926 07:25:25.285086 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true } } }
I20260926 07:25:25.285275 7160 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:25.285284 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: This master's current role is: LEARNER
I20260926 07:25:25.288638 7174 catalog_manager.cc:1259] Loaded cluster ID: 28dc622c9eb64edebd3dde2b34769ebe
I20260926 07:25:25.288789 7174 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: loading cluster ID for follower catalog manager: success
I20260926 07:25:25.291275 7174 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: acquiring CA information for follower catalog manager: success
I20260926 07:25:25.292840 7174 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:25:25.321856 7089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:25.322136 7089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:25.322213 7089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:25.326602 7089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:25.326678 7089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:25.326712 7089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:25.326766 7089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:25.326803 7089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:25.332139 7089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33023,127.0.98.61:38947,127.0.98.60:46361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:25.333083 7089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:25.334021 7089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:25.342105 7182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:25.342092 7180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:25.342130 7179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:25.342458 7089 server_base.cc:1034] running on GCE node
I20260926 07:25:25.343101 7089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:25.344070 7089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:25.345220 7089 hybrid_clock.cc:648] HybridClock initialized: now 1790407525345191 us; error 45 us; skew 500 ppm
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:25.347798 7089 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 07:25:25.348547 7089 webserver.cc:460] Webserver started at http://127.0.98.60:44479/ using document root <none> and password file <none>
I20260926 07:25:25.349028 7089 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:25.349083 7089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:25.349236 7089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:25.351112 7089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/data/instance:
uuid: "10b3febc784444d3967e52d3992e7c40"
format_stamp: "Formatted at 2026-09-26 07:25:25 on dist-test-slave-g04g"
server_key: "ad7547248742cdfe2a43bdbe81619bbe"
server_key_iv: "b339745569ca0a8a193b186524296738"
server_key_version: "encryptionkey@0"
I20260926 07:25:25.351589 7089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/wal/instance:
uuid: "10b3febc784444d3967e52d3992e7c40"
format_stamp: "Formatted at 2026-09-26 07:25:25 on dist-test-slave-g04g"
server_key: "ad7547248742cdfe2a43bdbe81619bbe"
server_key_iv: "b339745569ca0a8a193b186524296738"
server_key_version: "encryptionkey@0"
I20260926 07:25:25.355417 7089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 07:25:25.357594 7189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:25.358760 7089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:25.358906 7089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/wal
uuid: "10b3febc784444d3967e52d3992e7c40"
format_stamp: "Formatted at 2026-09-26 07:25:25 on dist-test-slave-g04g"
server_key: "ad7547248742cdfe2a43bdbe81619bbe"
server_key_iv: "b339745569ca0a8a193b186524296738"
server_key_version: "encryptionkey@0"
I20260926 07:25:25.359010 7089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:25.390125 7089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:25.393648 7089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:25.393832 7089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:25.401527 7089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46361
I20260926 07:25:25.401562 7241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46361 every 8 connection(s)
I20260926 07:25:25.402493 7089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/data/info.pb
I20260926 07:25:25.405403 7242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:25.408326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 7089
I20260926 07:25:25.409932 7242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 07:25:25.422746 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.411945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.0.98.62:33023)
0926 07:25:25.412087 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.412093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.412476 (+ 383us) server_negotiation.cc:407] Connection header received
0926 07:25:25.412582 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.412589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.412654 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.412748 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.413999 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.414716 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.416038 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.416364 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.416496 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.418805 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.418811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.418817 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.418852 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.420433 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.421005 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.421007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.421009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.421073 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.421418 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.421421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.421423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.421645 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.421717 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.422452 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.422577 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:25:25.424165 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.409898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50410 (local address 127.0.98.60:46361)
0926 07:25:25.410307 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.410321 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.410391 (+ 70us) server_negotiation.cc:407] Connection header received
0926 07:25:25.411019 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.411085 (+ 66us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.411439 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.411766 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.412691 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.414313 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.415373 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.416132 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.416236 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.419218 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.419226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.419239 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.419279 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.421168 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.421621 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.421625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.421626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.421813 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.422060 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.422063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.422065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.422299 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.422473 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.422751 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.423161 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
I20260926 07:25:25.424165 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.411776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:49340)
0926 07:25:25.412327 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.412370 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.412495 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.412797 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.412802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.413230 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.413840 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.413854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.414850 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.414852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.415916 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.415925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.416504 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.416507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.416625 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.417154 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.417170 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.418585 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.420536 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.420545 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.420558 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.420914 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.421157 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.421159 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.421162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.421328 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.421836 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.421845 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.422150 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.422978 (+ 828us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.423209 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":125,"thread_start_us":151,"threads_started":1}
I20260926 07:25:25.425526 7242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 07:25:25.433696 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.425997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38947 (local address 127.0.0.1:59376)
0926 07:25:25.426145 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.426159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.426230 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.426632 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.426635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.426795 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.426907 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.426913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.427856 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.427859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.428391 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.428398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.428782 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.428784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.428849 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.429253 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.429263 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.430443 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.432369 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.432372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.432373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.432578 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.432849 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.432851 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.432852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.432931 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.433276 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.433278 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.433355 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.433447 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.433550 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 07:25:25.433710 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.426100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.0.98.61:38947)
0926 07:25:25.426373 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.426378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.426392 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:25.426433 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.426436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.426480 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.426608 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.427011 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.427748 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.428483 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.428697 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.428771 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.430567 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.430570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.430573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.430595 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.432248 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.432688 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.432690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.432691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.432763 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.433030 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.433032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.433033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.433193 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.433266 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.433495 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.433593 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
I20260926 07:25:25.434514 7242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 07:25:25.442488 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.435021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:50414)
0926 07:25:25.435167 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.435180 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.435249 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.435474 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.435476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.435593 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.435656 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.435660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.436672 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.436674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.437158 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.437164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.437587 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.437589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.437641 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.438060 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.438070 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.439297 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.440858 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.440864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.440866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.441065 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.441356 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.441361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.441363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.441469 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.441840 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.441843 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.441934 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.442133 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.442298 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 07:25:25.442492 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.435111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.0.98.60:46361)
0926 07:25:25.435212 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.435215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.435228 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:25:25.435307 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.435309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.435343 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.435454 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.435747 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.436578 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.437268 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.437493 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.437578 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.439436 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.439446 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.439448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.439468 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.440679 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.441176 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.441179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.441181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.441259 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.441554 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.441561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.441563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.441738 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.441850 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.442196 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.442356 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":29}
I20260926 07:25:25.445389 7242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: Bootstrap starting.
I20260926 07:25:25.447701 6713 heartbeater.cc:503] Master 127.0.98.62:33023 was elected leader, sending a full tablet report...
I20260926 07:25:25.447721 7242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:25.448537 7242 log.cc:824] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:25.450394 7242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: No bootstrap required, opened a new log
I20260926 07:25:25.453322 7242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } }
I20260926 07:25:25.453519 7242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:25.453603 7242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b3febc784444d3967e52d3992e7c40, State: Initialized, Role: FOLLOWER
I20260926 07:25:25.453990 7242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } }
I20260926 07:25:25.454435 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } } }
I20260926 07:25:25.454566 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:25:25.455225 7242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:25:25.459741 7259 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:25:25.459817 7259 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:25:25.463948 7089 master_runner.cc:428] Detected that this master 10b3febc784444d3967e52d3992e7c40 is joining an existing cluster
I20260926 07:25:25.463999 7089 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:46361
I20260926 07:25:25.474893 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.466351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.0.98.62:33023)
0926 07:25:25.466480 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.466484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.466649 (+ 165us) server_negotiation.cc:407] Connection header received
0926 07:25:25.466817 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.466823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.466879 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.466934 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.467787 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.468621 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.469698 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.469929 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.470036 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.471597 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.471601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.471603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.471629 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.473266 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.473724 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.473727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.473729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.473821 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.474093 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.474099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.474100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.474270 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.474372 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.474633 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.474741 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:25:25.474903 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.465945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:49356)
0926 07:25:25.466553 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.466574 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.466695 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.467130 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.467132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.467459 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.467642 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.467650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.468733 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.468736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.469583 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.469596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.470048 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.470051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.470139 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.471108 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.471122 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.471462 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.473370 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.473373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.473376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.473628 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.473913 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.473917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.473919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.474005 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.474372 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.474387 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.474477 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.474580 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.474711 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":19,"thread_start_us":106,"threads_started":1}
I20260926 07:25:25.478035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 6584
I20260926 07:25:25.481189 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.466885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.0.98.60:46361)
0926 07:25:25.466984 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.466989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.472402 (+ 5413us) server_negotiation.cc:407] Connection header received
0926 07:25:25.472504 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.472507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.472551 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.472636 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.473132 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.473752 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.474835 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.475179 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.475246 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.476600 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.476604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.476606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.476628 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.478297 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.480048 (+ 1751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.480054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.480058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.480134 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.480398 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.480401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.480403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.480577 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.480680 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.480918 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.481062 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":40}
I20260926 07:25:25.481200 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.467124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:50426)
0926 07:25:25.472321 (+ 5197us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.472336 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.472429 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.472662 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.472665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.472900 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.473007 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.473013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.473868 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.473871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.474716 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.474726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.475578 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.475582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.475671 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.476059 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.476074 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.476468 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.479767 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.479769 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.479771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.479950 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.480220 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.480225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.480227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.480306 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.480669 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.480671 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.480774 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.480884 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.481010 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5127,"thread_start_us":89,"threads_started":1}
I20260926 07:25:25.485267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37491
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33023
--webserver_port=37607
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33023,127.0.98.61:38947
--tserver_master_addrs=127.0.98.62:33023,127.0.98.61:38947,127.0.98.60:46361 with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:25:25.492235 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.466056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38947 (local address 127.0.0.1:59388)
0926 07:25:25.466794 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.466807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.466934 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.467329 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.467332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.467570 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.467702 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.467710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.468484 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.468490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.469421 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.469430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.470158 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.470162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.470269 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.487649 (+ 17380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.487666 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.488114 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.490198 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.490205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.490209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.490501 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.490855 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.490859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.490862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.490965 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.491547 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.491552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.491684 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.491802 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.491964 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":4,"thread_start_us":529,"threads_started":1}
I20260926 07:25:25.492815 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.466143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.0.98.61:38947)
0926 07:25:25.466259 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.466266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.466866 (+ 600us) server_negotiation.cc:407] Connection header received
0926 07:25:25.467066 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.467072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.467125 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.467176 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.467855 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.468369 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.469558 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.469922 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.470032 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.488278 (+ 18246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.488283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.488286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.488320 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.490080 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.490635 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.490641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.490646 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.490739 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.491112 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.491117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.491120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.491391 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.491498 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.492453 (+ 955us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.492590 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 07:25:25.499356 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.489295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:50438)
0926 07:25:25.489428 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.489444 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.489530 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.490605 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.490609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.490791 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.490901 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.490908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.492012 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.492015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.492866 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.492876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.493583 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.493587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.493680 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.494208 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.494222 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.495915 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.497861 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.497867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.497871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.498102 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.498389 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.498392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.498394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.498479 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.498854 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.498857 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.498946 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.499070 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.499199 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 07:25:25.499389 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.490254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.0.98.60:46361)
0926 07:25:25.490356 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.490359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.490371 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:25:25.490431 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.490434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.490480 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.491007 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.491138 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.491886 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.493115 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.493460 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.494040 (+ 580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.496088 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.496092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.496094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.496117 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.497744 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.498207 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.498211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.498212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.498290 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.498569 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.498572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.498573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.498761 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.498853 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.499135 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.499246 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:25:25.500428 6522 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:46361: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "10b3febc784444d3967e52d3992e7c40"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 46361
}
attrs {
promote: true
}
}
dest_uuid: "46dc84ef76be4b5181e29f5cd68aacd5"
cas_config_opid_index: 5
I20260926 07:25:25.501011 6522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } }
I20260926 07:25:25.502067 7084 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 10b3febc784444d3967e52d3992e7c40 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } } }
I20260926 07:25:25.502921 7213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:25.503793 7084 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:46361
I20260926 07:25:25.503960 7084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } } }
I20260926 07:25:25.504045 7084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:25.504244 7213 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 FOLLOWER]: Refusing update from remote peer 46dc84ef76be4b5181e29f5cd68aacd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 07:25:25.504568 7089 master.cc:537] Master@127.0.98.60:46361 shutting down...
I20260926 07:25:25.505079 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:25:25.507019 6507 consensus_peers.cc:487] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 -> Peer 10b3febc784444d3967e52d3992e7c40 (127.0.98.60:46361): Couldn't send request to peer 10b3febc784444d3967e52d3992e7c40. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 07:25:25.509928 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.502017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38947 (local address 127.0.0.1:59404)
0926 07:25:25.505003 (+ 2986us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.505016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.505096 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.505753 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.505759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.505781 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:25.505886 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.505894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.506932 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.506935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.508306 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.508316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.509538 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.509542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.509634 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.509677 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.509746 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.509806 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2910}
I20260926 07:25:25.510423 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.502083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.0.98.61:38947)
0926 07:25:25.502197 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.502201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.505197 (+ 2996us) server_negotiation.cc:407] Connection header received
0926 07:25:25.505251 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.505273 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.505349 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.505458 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:25.506014 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.506812 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.508484 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.509142 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.509240 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.509392 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.510217 (+ 825us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.510291 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:25:25.510658 7089 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:25.510941 7089 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:25.511068 7089 tablet_replica.cc:330] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: stopping tablet replica
I20260926 07:25:25.511106 7134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 LEARNER]: Refusing update from remote peer 46dc84ef76be4b5181e29f5cd68aacd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 07:25:25.511494 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 07:25:25.513326 7134 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 10b3febc784444d3967e52d3992e7c40 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } } }
I20260926 07:25:25.515136 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46dc84ef76be4b5181e29f5cd68aacd5. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } } }
I20260926 07:25:25.515250 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: This master's current role is: LEARNER
I20260926 07:25:25.515333 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: true } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } } }
I20260926 07:25:25.515396 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: This master's current role is: LEARNER
I20260926 07:25:25.515548 7083 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5: attempting to promote NON_VOTER fd9da486a92542fcb2304179a2ab9f4c to VOTER
I20260926 07:25:25.516016 7083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } }
I20260926 07:25:25.517143 7134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 LEARNER]: Refusing update from remote peer 46dc84ef76be4b5181e29f5cd68aacd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260926 07:25:25.517149 6507 connection.cc:535] client connection to 127.0.98.60:46361 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:25:25.517458 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 07:25:25.517799 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.517551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:50442)
0926 07:25:25.517660 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.517727 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:25:25.518020 6507 consensus_peers.cc:487] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 -> Peer 10b3febc784444d3967e52d3992e7c40 (127.0.98.60:46361): Couldn't send request to peer 10b3febc784444d3967e52d3992e7c40. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 07:25:25.519129 7084 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fd9da486a92542fcb2304179a2ab9f4c (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } } }
I20260926 07:25:25.519567 7133 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fd9da486a92542fcb2304179a2ab9f4c (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } } }
I20260926 07:25:25.520483 7084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } } }
I20260926 07:25:25.520579 7084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:25.520741 7084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } } }
I20260926 07:25:25.520828 7084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:25.520990 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } } }
I20260926 07:25:25.521090 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:25:25.529145 7089 master.cc:552] Master@127.0.98.60:46361 shutdown complete.
I20260926 07:25:25.529197 7089 master_runner.cc:305] Clearing existing system tablet
I20260926 07:25:25.529460 7089 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:25.529515 7089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:25.530781 7089 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:25:25.531776 7272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:25.532002 7089 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:25.532132 7089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 07:25:25.532166 7089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/wal
uuid: "10b3febc784444d3967e52d3992e7c40"
format_stamp: "Formatted at 2026-09-26 07:25:25 on dist-test-slave-g04g"
server_key: "ad7547248742cdfe2a43bdbe81619bbe"
server_key_iv: "b339745569ca0a8a193b186524296738"
server_key_version: "encryptionkey@0"
I20260926 07:25:25.534076 7089 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:25.536322 7089 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 07:25:25.536405 7089 log.cc:1196] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 07:25:25.536633 7089 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: Deleting consensus metadata
I20260926 07:25:25.536852 7089 master_runner.cc:315] Copying system tablet from 127.0.98.62:33023
I20260926 07:25:25.537892 7089 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33023
I20260926 07:25:25.546902 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.538794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49372 (local address 127.0.98.62:33023)
0926 07:25:25.538907 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.538911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.538928 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:25:25.539011 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.539014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.539089 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.539178 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.539666 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.540462 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.541648 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.542000 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.542102 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.543604 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.543607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.543610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.543637 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.545238 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.545693 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.545697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.545699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.545784 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.546059 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.546062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.546064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.546259 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.546353 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.546622 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.546756 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:25:25.546905 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.538399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:49372)
0926 07:25:25.538817 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.538831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.538929 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.539207 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.539210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.539427 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.539547 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.539554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.540579 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.540583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.541538 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.541549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.542281 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.542286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.542382 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.543005 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.543020 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.543458 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.545345 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.545348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.545351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.545595 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.545875 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.545878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.545880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.545969 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.546353 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.546356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.546451 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.546558 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.546704 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"spinlock_wait_cycles":126208,"thread_start_us":141,"threads_started":1}
I20260926 07:25:25.547698 6542 tablet_copy_service.cc:140] P 46dc84ef76be4b5181e29f5cd68aacd5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 10b3febc784444d3967e52d3992e7c40 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49372)
I20260926 07:25:25.547791 6542 tablet_copy_service.cc:161] P 46dc84ef76be4b5181e29f5cd68aacd5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 10b3febc784444d3967e52d3992e7c40 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49372: session id = 10b3febc784444d3967e52d3992e7c40-00000000000000000000000000000000
I20260926 07:25:25.548790 6542 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 07:25:25.550010 7089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 07:25:25.552246 7089 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: tablet copy: Starting download of 0 data blocks...
I20260926 07:25:25.552505 7089 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: tablet copy: Starting download of 1 WAL segments...
I20260926 07:25:25.553972 7089 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 07:25:25.555722 6542 tablet_copy_service.cc:342] P 46dc84ef76be4b5181e29f5cd68aacd5: Request end of tablet copy session 10b3febc784444d3967e52d3992e7c40-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49372
I20260926 07:25:25.555791 6542 tablet_copy_service.cc:433] P 46dc84ef76be4b5181e29f5cd68aacd5: ending tablet copy session 10b3febc784444d3967e52d3992e7c40-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 10b3febc784444d3967e52d3992e7c40
I20260926 07:25:25.563213 7089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:25.564981 7282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:25.564988 7279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:25.565522 7089 server_base.cc:1034] running on GCE node
W20260926 07:25:25.567457 7280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:25.567657 7089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:25.567929 7089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:25.569094 7089 hybrid_clock.cc:648] HybridClock initialized: now 1790407525569080 us; error 29 us; skew 500 ppm
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:25.570619 7089 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 07:25:25.571126 7089 webserver.cc:460] Webserver started at http://127.0.98.60:41763/ using document root <none> and password file <none>
I20260926 07:25:25.571311 7089 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:25.571352 7089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:25.572572 7089 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:25.573506 7289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:25.573763 7089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:25.573845 7089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/wal
uuid: "10b3febc784444d3967e52d3992e7c40"
format_stamp: "Formatted at 2026-09-26 07:25:25 on dist-test-slave-g04g"
server_key: "ad7547248742cdfe2a43bdbe81619bbe"
server_key_iv: "b339745569ca0a8a193b186524296738"
server_key_version: "encryptionkey@0"
I20260926 07:25:25.573922 7089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:25.601221 7089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:25.603395 7089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:25.608381 7089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46361
I20260926 07:25:25.608420 7341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46361 every 8 connection(s)
I20260926 07:25:25.608973 7089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/master-2/data/info.pb
I20260926 07:25:25.610720 7342 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:25:25.611799 7342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: Bootstrap starting.
I20260926 07:25:25.622596 7342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:25:25.622907 7342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: Bootstrap complete.
I20260926 07:25:25.623638 7342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } }
I20260926 07:25:25.623766 7342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10b3febc784444d3967e52d3992e7c40, State: Initialized, Role: LEARNER
I20260926 07:25:25.623862 7342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } }
I20260926 07:25:25.624276 7342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:25.626920 7348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } } }
I20260926 07:25:25.627074 7348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: This master's current role is: LEARNER
I20260926 07:25:25.628229 7359 catalog_manager.cc:1259] Loaded cluster ID: 28dc622c9eb64edebd3dde2b34769ebe
W20260926 07:25:25.627995 7268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:25.628314 7268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:25.628412 7268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 07:25:25.628409 7359 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: loading cluster ID for follower catalog manager: success
I20260926 07:25:25.630563 7359 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: acquiring CA information for follower catalog manager: success
I20260926 07:25:25.631974 7359 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:25:25.633708 7268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:25.633795 7268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:25.633908 7268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:25.639415 7268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37607
--tserver_master_addrs=127.0.98.62:33023,127.0.98.61:38947,127.0.98.60:46361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:25.640429 7268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:25.641440 7268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:25.651127 7362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:25.651100 7361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:25.651297 7268 server_base.cc:1034] running on GCE node
W20260926 07:25:25.651070 7364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:25.651824 7268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:25.652390 7268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:25.653471 6847 heartbeater.cc:503] Master 127.0.98.62:33023 was elected leader, sending a full tablet report...
I20260926 07:25:25.653558 7268 hybrid_clock.cc:648] HybridClock initialized: now 1790407525653522 us; error 30 us; skew 500 ppm
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:25.656553 7268 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 07:25:25.657410 7268 webserver.cc:460] Webserver started at http://127.0.98.1:37607/ using document root <none> and password file <none>
I20260926 07:25:25.657953 7268 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:25.658067 7268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:25.661737 7268 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:25.663765 7371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:25.664940 7268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:25.665067 7268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "852fb4b9b0fb4ed187c9f283d7bc3132"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "53f99c334c545bd58288ea2dad9bbe5e"
server_key_iv: "bcea670fc16ec03e39ff3edf187b69bf"
server_key_version: "encryptionkey@0"
I20260926 07:25:25.665570 7268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:25.682715 7268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:25.685750 7268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:25.685945 7268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:25.686523 7268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:25.687533 7268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:25.687598 7268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:25.687659 7268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:25.687690 7268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:25.697158 7268 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37491
I20260926 07:25:25.697181 7484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37491 every 8 connection(s)
I20260926 07:25:25.698101 7268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:25.703424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 7268
I20260926 07:25:25.703604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 6718
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:25.710173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36701
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33023
--webserver_port=34931
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33023,127.0.98.61:38947
--tserver_master_addrs=127.0.98.62:33023,127.0.98.61:38947,127.0.98.60:46361 with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 07:25:25.716939 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.700057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35063 (local address 127.0.98.61:38947)
0926 07:25:25.700507 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.700513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.701142 (+ 629us) server_negotiation.cc:407] Connection header received
0926 07:25:25.702354 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.702358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.702424 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.702480 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.705400 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.705952 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.706930 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.707412 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.707536 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.711186 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.711193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.711200 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.711237 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.713759 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.714732 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.714739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.714744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.714838 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.715572 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.715578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.715581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.715862 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.715955 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.716510 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.716686 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260926 07:25:25.717775 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.700329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.1:52451)
0926 07:25:25.700902 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.700967 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.702118 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.702684 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.702687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.703152 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.703971 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.703994 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.705399 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.705403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.706807 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.706817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.707519 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.707523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.707728 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.708561 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.708581 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.710935 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.713864 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.713872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.713891 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.714291 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.714880 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.714884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.714887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.715379 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.715885 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.715892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.716202 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.716386 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.716724 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":114,"spinlock_wait_cycles":14336,"thread_start_us":141,"threads_started":1}
I20260926 07:25:25.718060 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.700344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38947 (local address 127.0.98.1:35063)
0926 07:25:25.701060 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.701073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.702114 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.702681 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.702685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.703155 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.705233 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.705241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.706074 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.706081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.706801 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.706810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.707680 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.707684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.708235 (+ 551us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.708767 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.708784 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.710966 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.713907 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.713913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.713919 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.714603 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.714967 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.714973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.714978 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.715432 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.716045 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.716050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.716214 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.717168 (+ 954us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.717369 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":233,"spinlock_wait_cycles":105472,"thread_start_us":117,"threads_started":1}
I20260926 07:25:25.718343 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.700473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44949 (local address 127.0.98.60:46361)
0926 07:25:25.700912 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.700924 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.701784 (+ 860us) server_negotiation.cc:407] Connection header received
0926 07:25:25.702325 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.702329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.702388 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.702496 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.708932 (+ 6436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.710036 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.711168 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.711559 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.711671 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.714125 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.714127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.714129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.714150 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.715879 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.716300 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.716303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.716304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.716395 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.716658 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.716661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.716662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.716844 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.716927 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.717879 (+ 952us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.717994 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":90,"threads_started":1}
I20260926 07:25:25.718715 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.700721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52451 (local address 127.0.98.62:33023)
0926 07:25:25.701338 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.701341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.701361 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:25:25.702438 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.702442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.702500 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.702574 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.704220 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.705240 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.706947 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.707263 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.707351 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.711156 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.711159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.711162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.711189 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.713729 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.714653 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.714657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.714659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.714749 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.715514 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.715517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.715520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.715748 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.718375 (+ 2627us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.718460 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.718570 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524}
I20260926 07:25:25.717787 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.700329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.98.1:44949)
0926 07:25:25.701674 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.701689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.702122 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.702673 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.702679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.703148 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.708764 (+ 5616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.708776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.710174 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.710178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.711017 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.711051 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.711819 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.711826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.711931 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.712315 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.712336 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.713977 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.715975 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.715977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.715979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.716193 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.716485 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.716488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.716490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.716565 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.716966 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.716969 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.717057 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.717149 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.717276 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1248,"mutex_wait_us":1079,"spinlock_wait_cycles":22784,"thread_start_us":109,"threads_started":1}
I20260926 07:25:25.719368 7486 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46361
I20260926 07:25:25.719359 7485 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33023
I20260926 07:25:25.719360 7487 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38947
I20260926 07:25:25.719630 7487 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:25.719627 7485 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:25.719627 7486 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:25.720207 7486 heartbeater.cc:511] Master 127.0.98.60:46361 requested a full tablet report, sending...
I20260926 07:25:25.720207 7485 heartbeater.cc:511] Master 127.0.98.62:33023 requested a full tablet report, sending...
I20260926 07:25:25.720227 7487 heartbeater.cc:511] Master 127.0.98.61:38947 requested a full tablet report, sending...
I20260926 07:25:25.721401 6522 ts_manager.cc:194] Re-registered known tserver with Master: 852fb4b9b0fb4ed187c9f283d7bc3132 (127.0.98.1:37491)
I20260926 07:25:25.721637 7298 ts_manager.cc:194] Registered new tserver with Master: 852fb4b9b0fb4ed187c9f283d7bc3132 (127.0.98.1:37491)
I20260926 07:25:25.721834 7124 ts_manager.cc:194] Registered new tserver with Master: 852fb4b9b0fb4ed187c9f283d7bc3132 (127.0.98.1:37491)
I20260926 07:25:25.722079 6522 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52451
W20260926 07:25:25.837635 7493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:25.837940 7493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:25.838043 7493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:25.842579 7493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:25.842658 7493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:25.842751 7493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:25.848625 7493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34931
--tserver_master_addrs=127.0.98.62:33023,127.0.98.61:38947,127.0.98.60:46361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:25.849709 7493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:25.850729 7493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:25:25.860322 7493 server_base.cc:1034] running on GCE node
W20260926 07:25:25.860281 7498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:25.860265 7501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:25.860260 7499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:25.860922 7493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:25.861465 7493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:25.862623 7493 hybrid_clock.cc:648] HybridClock initialized: now 1790407525862607 us; error 29 us; skew 500 ppm
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:25.865299 7493 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 07:25:25.866175 7493 webserver.cc:460] Webserver started at http://127.0.98.2:34931/ using document root <none> and password file <none>
I20260926 07:25:25.866716 7493 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:25.866817 7493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:25.870251 6981 heartbeater.cc:503] Master 127.0.98.62:33023 was elected leader, sending a full tablet report...
I20260926 07:25:25.870810 7493 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:25.872818 7508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:25.873961 7493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:25.874074 7493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "ef7a5385aae84f3192e11aa9e82c233a"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "4e16f63ddcdb4d2fd5929aeb2312980d"
server_key_iv: "5eb82c47232dbe657d4d00badaa2ab70"
server_key_version: "encryptionkey@0"
I20260926 07:25:25.874629 7493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:25.915486 7493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:25.918331 7493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:25.918529 7493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:25.919191 7493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:25.920032 7493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:25.920076 7493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:25.920145 7493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:25.920176 7493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:25.929922 7493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36701
I20260926 07:25:25.929948 7621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36701 every 8 connection(s)
I20260926 07:25:25.930920 7493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:25:25.938618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 7493
I20260926 07:25:25.938782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 6852
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 07:25:25.945987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34843
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33023
--webserver_port=36135
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33023,127.0.98.61:38947
--tserver_master_addrs=127.0.98.62:33023,127.0.98.61:38947,127.0.98.60:46361 with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:25:25 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:25.948012 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.933999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38247 (local address 127.0.98.60:46361)
0926 07:25:25.934113 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.934117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.934260 (+ 143us) server_negotiation.cc:407] Connection header received
0926 07:25:25.935378 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.935381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.935441 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.935517 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.936800 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.937327 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.938558 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.938924 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.939018 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.943107 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.943111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.943114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.943144 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.945359 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.946022 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.946025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.946027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.946252 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.946649 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.946653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.946655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.946860 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.946934 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.947728 (+ 794us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.947850 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:25:25.948012 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.932717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49647 (local address 127.0.98.61:38947)
0926 07:25:25.932865 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.932870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.934259 (+ 1389us) server_negotiation.cc:407] Connection header received
0926 07:25:25.935396 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.935399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.935449 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.935502 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.936807 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.937594 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.940387 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.940687 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.940818 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.943479 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.943485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.943490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.943528 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.945254 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.946021 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.946025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.946027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.946130 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.946649 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.946653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.946655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.946870 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.946945 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.947728 (+ 783us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.947857 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:25:25.949694 7627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.933089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.98.2:38247)
0926 07:25:25.934114 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.934169 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.935187 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.935705 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.935707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.936128 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.936660 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.936673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.937471 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.937474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.938416 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.938428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.939810 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.939814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.939966 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.940622 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.940648 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.942913 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.945465 (+ 2552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.945470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.945475 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.945906 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.946393 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.946398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.946402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.946540 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.947012 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.947019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.947443 (+ 424us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.948547 (+ 1104us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.948650 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":110,"spinlock_wait_cycles":49792,"thread_start_us":137,"threads_started":1}
I20260926 07:25:25.949985 7628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.933089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38947 (local address 127.0.98.2:49647)
0926 07:25:25.934115 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.934169 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.935173 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.935704 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.935707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.936132 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.936670 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.936679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.937732 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.937735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.938842 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.940264 (+ 1422us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.940819 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.940823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.940917 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.941403 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.941422 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.943327 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.945389 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.945398 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.945419 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.945906 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.946347 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.946353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.946358 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.946540 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.947012 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.947019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.947443 (+ 424us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.948230 (+ 787us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.948586 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":336,"spinlock_wait_cycles":92800,"thread_start_us":132,"threads_started":1}
I20260926 07:25:25.950559 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.933454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42955 (local address 127.0.98.62:33023)
0926 07:25:25.933563 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:25.933567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:25.934368 (+ 801us) server_negotiation.cc:407] Connection header received
0926 07:25:25.935396 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:25.935399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:25.935449 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:25.935502 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:25.936808 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.937601 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.943658 (+ 6057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.944124 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.944216 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.946749 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:25.946755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:25.946761 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:25.946794 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:25.948653 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.949177 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.949183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.949188 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.949274 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:25.949577 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:25.949582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:25.949587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:25.949833 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:25.949911 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:25.950296 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 07:25:25.950418 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:25:25.951081 7626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:25.933782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.2:42955)
0926 07:25:25.934291 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:25.934303 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:25.935173 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:25.935696 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:25.935706 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:25.936137 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:25.936672 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.936679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.937732 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.937735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.938824 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:25.938834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:25.944405 (+ 5571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:25.944411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:25.944533 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:25.945071 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:25.945090 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:25.946576 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:25.948804 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.948810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.948815 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.949064 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.949384 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:25.949390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:25.949395 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:25.949481 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:25.949989 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:25.949995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:25.950103 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 07:25:25.950754 (+ 651us) client_negotiation.cc:240] Negotiation successful
0926 07:25:25.950893 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":175,"spinlock_wait_cycles":67328,"thread_start_us":106,"threads_started":1}
I20260926 07:25:25.951241 7623 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46361
I20260926 07:25:25.951327 7624 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38947
I20260926 07:25:25.951491 7623 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:25.951493 7624 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:25.951972 7622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33023
I20260926 07:25:25.952036 7622 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:25.952126 7623 heartbeater.cc:511] Master 127.0.98.60:46361 requested a full tablet report, sending...
I20260926 07:25:25.952164 7622 heartbeater.cc:511] Master 127.0.98.62:33023 requested a full tablet report, sending...
I20260926 07:25:25.952708 7624 heartbeater.cc:511] Master 127.0.98.61:38947 requested a full tablet report, sending...
I20260926 07:25:25.953321 6522 ts_manager.cc:194] Re-registered known tserver with Master: ef7a5385aae84f3192e11aa9e82c233a (127.0.98.2:36701)
I20260926 07:25:25.953370 7124 ts_manager.cc:194] Registered new tserver with Master: ef7a5385aae84f3192e11aa9e82c233a (127.0.98.2:36701)
I20260926 07:25:25.953370 7298 ts_manager.cc:194] Registered new tserver with Master: ef7a5385aae84f3192e11aa9e82c233a (127.0.98.2:36701)
I20260926 07:25:25.954192 6522 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42955
W20260926 07:25:26.057179 7629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:26.057405 7629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:26.057446 7629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:26.061255 7629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:26.061303 7629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:26.061363 7629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:26.066299 7629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36135
--tserver_master_addrs=127.0.98.62:33023,127.0.98.61:38947,127.0.98.60:46361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:26.067286 7629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:26.068208 7629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:26.077895 7634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:26.078013 7629 server_base.cc:1034] running on GCE node
W20260926 07:25:26.077895 7637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:26.077926 7635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:26.078616 7629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:26.079311 7629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:26.080504 7629 hybrid_clock.cc:648] HybridClock initialized: now 1790407526080451 us; error 68 us; skew 500 ppm
Sep 26 07:25:26 dist-test-slave-g04g krb5kdc[6477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:26.083292 7629 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 07:25:26.084160 7629 webserver.cc:460] Webserver started at http://127.0.98.3:36135/ using document root <none> and password file <none>
I20260926 07:25:26.084702 7629 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:26.084802 7629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:26.088912 7629 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:26.090963 7644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:26.092710 7629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:25:26.092854 7629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "c08cb6a0209647a49169edae08439ffc"
format_stamp: "Formatted at 2026-09-26 07:25:24 on dist-test-slave-g04g"
server_key: "797038a272ff2b29e7a46a1f1158e0a3"
server_key_iv: "9e54d8a8865d4f01cc0d231013d13aaa"
server_key_version: "encryptionkey@0"
I20260926 07:25:26.093429 7629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:26.095310 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.091293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:50446)
0926 07:25:26.091531 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:26.091544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:26.091643 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:26.091991 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:26.091994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:26.092013 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:26.092107 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.092111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.093051 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.093054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.093940 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.093949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.094873 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.094876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.094982 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.095013 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 07:25:26.095120 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:25:26.095195 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":62,"threads_started":1}
I20260926 07:25:26.095314 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.091422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.0.98.60:46361)
0926 07:25:26.091544 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.091549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.091599 (+ 50us) server_negotiation.cc:407] Connection header received
0926 07:25:26.091785 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.091788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.091869 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.091968 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:26.092207 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.092947 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.094058 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.094751 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.094862 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.094994 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.095135 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.095201 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:25:26.095906 7312 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 07:25:26.096261 7348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46dc84ef76be4b5181e29f5cd68aacd5. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } } }
I20260926 07:25:26.096398 7348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: This master's current role is: LEARNER
I20260926 07:25:26.115623 7629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:26.118472 7629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:26.118664 7629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:26.119298 7629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:26.120139 7629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:26.120183 7629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:26.120222 7629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:26.120275 7629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:26.129815 7629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34843
I20260926 07:25:26.130113 7758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34843 every 8 connection(s)
I20260926 07:25:26.130961 7629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:26.133132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 7629
Sep 26 07:25:26 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 07:25:26 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:26.146876 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.136063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.0.98.60:46361)
0926 07:25:26.136759 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.136763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.136778 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:25:26.136815 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.136818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.136861 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.136951 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:26.137837 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.138532 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.139760 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.140102 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.140173 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.142531 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:26.142535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:26.142538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:26.142563 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:26.144459 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.144981 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.144985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.144987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.145070 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.145388 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.145392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.145394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.145706 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:26.145811 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.146547 (+ 736us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.146691 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"spinlock_wait_cycles":2816,"thread_start_us":77,"threads_started":1}
I20260926 07:25:26.147828 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.132808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47795 (local address 127.0.98.60:46361)
0926 07:25:26.132973 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.132979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.133913 (+ 934us) server_negotiation.cc:407] Connection header received
0926 07:25:26.135486 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.135490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.135552 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.135661 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:26.137647 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.138486 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.140129 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.140427 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.140504 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.143844 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:26.143848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:26.143853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:26.143880 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:26.145490 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.146107 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.146111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.146114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.146206 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.146607 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.146610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.146611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.146792 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:26.146865 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.147579 (+ 714us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.147695 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"spinlock_wait_cycles":1792}
I20260926 07:25:26.148844 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.133284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.98.3:47795)
0926 07:25:26.133752 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:26.133804 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:26.134938 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:26.135851 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:26.135858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:26.136715 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:26.137473 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.137497 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.138669 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.138672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.140008 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.140017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.140687 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.140689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.140898 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.141450 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:26.141470 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:26.143650 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:26.145602 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:26.145608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:26.145623 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:26.145995 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:26.146319 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:26.146323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:26.146325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:26.146504 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:26.146935 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:26.146943 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:26.147296 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 07:25:26.147527 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 07:25:26.147839 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":11,"thread_start_us":136,"threads_started":1}
I20260926 07:25:26.150098 7760 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46361
I20260926 07:25:26.150323 7760 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:26.150889 7760 heartbeater.cc:511] Master 127.0.98.60:46361 requested a full tablet report, sending...
Sep 26 07:25:26 dist-test-slave-g04g krb5kdc[6477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 07:25:26.152127 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.136333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.0.98.61:38947)
0926 07:25:26.136597 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.136601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.137786 (+ 1185us) server_negotiation.cc:407] Connection header received
0926 07:25:26.137833 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.137838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.137888 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.137968 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:26.138831 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.139616 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.140761 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.141098 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.141198 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.145691 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:26.145698 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:26.145703 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:26.145752 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:26.147639 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.150461 (+ 2822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.150464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.150466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.150558 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.150876 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.150880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.150881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.151237 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:26.151368 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.151771 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.151912 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
I20260926 07:25:26.152864 7298 ts_manager.cc:194] Registered new tserver with Master: c08cb6a0209647a49169edae08439ffc (127.0.98.3:34843)
I20260926 07:25:26.154458 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.134887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48747 (local address 127.0.98.62:33023)
0926 07:25:26.135183 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.135188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.141012 (+ 5824us) server_negotiation.cc:407] Connection header received
0926 07:25:26.141178 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.141182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.141242 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.141300 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:26.141945 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.142713 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.143996 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.144383 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.144469 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.146964 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:26.146969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:26.146976 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:26.147009 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:26.149740 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.152750 (+ 3010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.152754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.152756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.152843 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.153190 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.153194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.153196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.153403 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:26.153494 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.154174 (+ 680us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.154314 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260926 07:25:26.154481 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.134763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.3:48747)
0926 07:25:26.140930 (+ 6167us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:26.140946 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:26.141074 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:26.141443 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:26.141446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:26.141701 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:26.141809 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.141816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.142884 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.142890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.143858 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.143871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.144636 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.144640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.144751 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.145203 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:26.145222 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:26.146793 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:26.149835 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:26.149838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:26.149840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:26.152631 (+ 2791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:26.152959 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:26.152966 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:26.152969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:26.153066 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:26.153899 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:26.153902 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:26.153996 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 07:25:26.154105 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:25:26.154266 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6086,"thread_start_us":89,"threads_started":1}
I20260926 07:25:26.155069 7759 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33023
I20260926 07:25:26.155161 7759 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:26.155349 7759 heartbeater.cc:511] Master 127.0.98.62:33023 requested a full tablet report, sending...
I20260926 07:25:26.155484 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.134481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35283 (local address 127.0.98.61:38947)
0926 07:25:26.134602 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.134607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.135566 (+ 959us) server_negotiation.cc:407] Connection header received
0926 07:25:26.135779 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.135783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.135838 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.135901 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:26.139302 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.140014 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.147791 (+ 7777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.148082 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.148175 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.151467 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:26.151471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:26.151473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:26.151494 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:26.153112 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.153883 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.153886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.153888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.153968 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.154262 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.154267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.154272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.154457 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:26.154542 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.155244 (+ 702us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.155353 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:25:26.155942 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.134360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38947 (local address 127.0.98.3:35283)
0926 07:25:26.135467 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:26.135486 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:26.135640 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:26.136056 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:26.136058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:26.136719 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:26.138176 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.138187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.140145 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.140148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.141844 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.141855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.148709 (+ 6854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.148713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.148850 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.149917 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:26.149929 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:26.151357 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:26.153527 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:26.153530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:26.153532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:26.153770 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:26.154066 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:26.154069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:26.154071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:26.154157 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:26.154808 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:26.154811 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:26.155025 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 07:25:26.155645 (+ 620us) client_negotiation.cc:240] Negotiation successful
0926 07:25:26.155788 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":880,"spinlock_wait_cycles":1920,"thread_start_us":125,"threads_started":1}
I20260926 07:25:26.155928 6522 ts_manager.cc:194] Re-registered known tserver with Master: c08cb6a0209647a49169edae08439ffc (127.0.98.3:34843)
I20260926 07:25:26.156454 7761 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38947
I20260926 07:25:26.156528 7761 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:26.156674 7761 heartbeater.cc:511] Master 127.0.98.61:38947 requested a full tablet report, sending...
I20260926 07:25:26.156638 6522 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48747
I20260926 07:25:26.157087 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.135265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.0.98.62:33023)
0926 07:25:26.137157 (+ 1892us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.137162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.137176 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:26.137216 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.137218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.137274 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.137371 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:26.138308 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.139162 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.149266 (+ 10104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.149653 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.149775 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.153272 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:26.153275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:26.153278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:26.153298 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:26.155222 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.155734 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.155737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.155739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.155832 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.156128 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.156131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.156133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.156340 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:26.156475 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.156801 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.156917 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1815,"thread_start_us":96,"threads_started":1}
I20260926 07:25:26.157262 7124 ts_manager.cc:194] Registered new tserver with Master: c08cb6a0209647a49169edae08439ffc (127.0.98.3:34843)
I20260926 07:25:26.160616 6522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:25:26.163852 6522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:26.166252 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: true } } }
I20260926 07:25:26.166370 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:26.169863 7778 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5: attempting to promote NON_VOTER 10b3febc784444d3967e52d3992e7c40 to VOTER
I20260926 07:25:26.170168 7778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } }
I20260926 07:25:26.171175 7133 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 FOLLOWER]: Refusing update from remote peer 46dc84ef76be4b5181e29f5cd68aacd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 07:25:26.171594 7778 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 07:25:26.171685 7311 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 LEARNER]: Refusing update from remote peer 46dc84ef76be4b5181e29f5cd68aacd5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 07:25:26.172065 7778 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 07:25:26.173082 7785 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 10b3febc784444d3967e52d3992e7c40 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } } }
I20260926 07:25:26.173419 7133 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 10b3febc784444d3967e52d3992e7c40 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } } }
I20260926 07:25:26.174517 7777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } } }
I20260926 07:25:26.174546 7785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } } }
I20260926 07:25:26.174633 7777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:26.174685 7785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:26.176469 7777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } } }
I20260926 07:25:26.176568 7777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:26.176774 7311 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 10b3febc784444d3967e52d3992e7c40 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } } }
I20260926 07:25:26.178620 7348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } } }
I20260926 07:25:26.178742 7348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:25:26.178937 7787 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } } }
I20260926 07:25:26.179075 7787 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:25:26.185115 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.180039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37491 (local address 127.0.0.1:54058)
0926 07:25:26.180143 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:26.180159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:26.180277 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:26.181285 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:26.181288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:26.181304 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:26.181394 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.181400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.182745 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.182749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.183637 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.183646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.184753 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.184757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.184844 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.184870 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:25:26.184921 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 07:25:26.184986 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 07:25:26.185763 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.180315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.0.98.1:37491)
0926 07:25:26.180793 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.180808 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.180836 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:25:26.180913 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.180918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.181133 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.181320 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:26.181486 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.182602 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.183782 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.184612 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.185342 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.185492 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.185540 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.185622 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":86,"threads_started":1}
I20260926 07:25:26.185840 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.180348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36701 (local address 127.0.0.1:43946)
0926 07:25:26.180880 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:26.180893 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:26.180980 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:26.181846 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:26.181852 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:26.181870 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:26.181977 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.181984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.183461 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.183465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.184279 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.184287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.185463 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.185466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.185561 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.185582 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:25:26.185642 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 07:25:26.185718 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":87,"threads_started":1}
I20260926 07:25:26.185978 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.181021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.0.98.2:36701)
0926 07:25:26.181352 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.181364 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.181387 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:25:26.181464 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.181469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.181674 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.182148 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:26.182293 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.183351 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.184399 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.185355 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.185466 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.185604 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.185669 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.185748 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":133,"threads_started":1}
I20260926 07:25:26.188901 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.180645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34843 (local address 127.0.0.1:44128)
0926 07:25:26.182106 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:26.182118 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:26.182198 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:26.182605 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:26.182609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:26.182626 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:26.182721 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.182726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.184541 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.184545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.186849 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.186859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.188545 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.188549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.188632 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.188653 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:25:26.188722 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:25:26.188786 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1397,"thread_start_us":111,"threads_started":1}
I20260926 07:25:26.188814 7556 tablet_service.cc:1467] Processing CreateTablet for tablet 21eb6358f3dd43a3abf56eb3f597f9c8 (DEFAULT_TABLE table=test-table [id=10d98466daef4e088d4acd65ac336e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:26.189411 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.180738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.0.98.3:34843)
0926 07:25:26.181088 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.181099 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.182191 (+ 1092us) server_negotiation.cc:407] Connection header received
0926 07:25:26.182305 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.182310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.182465 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.182601 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:26.182855 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.184410 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.186989 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.188261 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.188364 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.188485 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.189159 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.189264 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":91,"threads_started":1}
I20260926 07:25:26.189293 7419 tablet_service.cc:1467] Processing CreateTablet for tablet 21eb6358f3dd43a3abf56eb3f597f9c8 (DEFAULT_TABLE table=test-table [id=10d98466daef4e088d4acd65ac336e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:26.189868 7556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21eb6358f3dd43a3abf56eb3f597f9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:26.190158 7419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21eb6358f3dd43a3abf56eb3f597f9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:26.192015 7693 tablet_service.cc:1467] Processing CreateTablet for tablet 21eb6358f3dd43a3abf56eb3f597f9c8 (DEFAULT_TABLE table=test-table [id=10d98466daef4e088d4acd65ac336e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:26.192924 7693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21eb6358f3dd43a3abf56eb3f597f9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:26.196892 7793 tablet_bootstrap.cc:492] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a: Bootstrap starting.
I20260926 07:25:26.198125 7794 tablet_bootstrap.cc:492] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132: Bootstrap starting.
I20260926 07:25:26.199922 7793 tablet_bootstrap.cc:654] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:26.200284 7794 tablet_bootstrap.cc:654] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:26.200783 7793 log.cc:824] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:26.201015 7794 log.cc:824] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:26.201023 7795 tablet_bootstrap.cc:492] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc: Bootstrap starting.
I20260926 07:25:26.202921 7794 tablet_bootstrap.cc:492] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132: No bootstrap required, opened a new log
I20260926 07:25:26.202997 7793 tablet_bootstrap.cc:492] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a: No bootstrap required, opened a new log
I20260926 07:25:26.203061 7795 tablet_bootstrap.cc:654] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:26.203172 7794 ts_tablet_manager.cc:1397] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:25:26.203210 7793 ts_tablet_manager.cc:1397] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:25:26.203807 7795 log.cc:824] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:26.205536 7795 tablet_bootstrap.cc:492] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc: No bootstrap required, opened a new log
I20260926 07:25:26.205694 7795 ts_tablet_manager.cc:1397] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 07:25:26.207453 7794 raft_consensus.cc:348] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } }
I20260926 07:25:26.207677 7794 raft_consensus.cc:374] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:26.207738 7794 raft_consensus.cc:729] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852fb4b9b0fb4ed187c9f283d7bc3132, State: Initialized, Role: FOLLOWER
I20260926 07:25:26.207891 7793 raft_consensus.cc:348] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } }
I20260926 07:25:26.208137 7793 raft_consensus.cc:374] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:26.208194 7793 raft_consensus.cc:729] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef7a5385aae84f3192e11aa9e82c233a, State: Initialized, Role: FOLLOWER
I20260926 07:25:26.208218 7794 consensus_queue.cc:260] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } }
I20260926 07:25:26.208644 7793 consensus_queue.cc:260] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } }
I20260926 07:25:26.209682 7793 ts_tablet_manager.cc:1428] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:25:26.209995 7622 heartbeater.cc:503] Master 127.0.98.62:33023 was elected leader, sending a full tablet report...
I20260926 07:25:26.210064 7795 raft_consensus.cc:348] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } }
I20260926 07:25:26.210279 7795 raft_consensus.cc:374] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:26.210323 7795 raft_consensus.cc:729] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08cb6a0209647a49169edae08439ffc, State: Initialized, Role: FOLLOWER
I20260926 07:25:26.210692 7795 consensus_queue.cc:260] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } }
I20260926 07:25:26.211005 7485 heartbeater.cc:503] Master 127.0.98.62:33023 was elected leader, sending a full tablet report...
I20260926 07:25:26.211592 7795 ts_tablet_manager.cc:1428] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:25:26.212455 7794 ts_tablet_manager.cc:1428] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260926 07:25:26.212646 7759 heartbeater.cc:503] Master 127.0.98.62:33023 was elected leader, sending a full tablet report...
I20260926 07:25:26.320091 7800 raft_consensus.cc:482] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:26.320345 7800 raft_consensus.cc:504] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } }
I20260926 07:25:26.321506 7800 leader_election.cc:288] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 852fb4b9b0fb4ed187c9f283d7bc3132 (127.0.98.1:37491), c08cb6a0209647a49169edae08439ffc (127.0.98.3:34843)
I20260926 07:25:26.325500 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.322000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35029 (local address 127.0.98.3:34843)
0926 07:25:26.322119 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.322125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.322213 (+ 88us) server_negotiation.cc:407] Connection header received
0926 07:25:26.322359 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.322362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.322409 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.322490 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:26.322742 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.323293 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.324417 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.324904 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.325020 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.325078 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.325297 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.325369 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:25:26.325516 7628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.321776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34843 (local address 127.0.98.2:35029)
0926 07:25:26.322117 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:26.322132 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:26.322235 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:26.322499 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:26.322501 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:26.322518 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:26.322632 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.322637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.323410 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.323416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.324298 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.324308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.325023 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.325027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.325166 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.325185 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 07:25:26.325269 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 07:25:26.325339 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":197}
I20260926 07:25:26.325557 7626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.321776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37491 (local address 127.0.98.2:46611)
0926 07:25:26.321926 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:26.321958 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:26.322071 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:26.322329 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:26.322332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:26.322347 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:26.322467 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.322472 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.323344 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.323347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.324286 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:26.324295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.325027 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.325031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:26.325221 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.325239 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:25:26.325324 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 07:25:26.325390 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":45}
I20260926 07:25:26.326092 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.321899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46611 (local address 127.0.98.1:37491)
0926 07:25:26.322039 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.322044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.322059 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:25:26.322172 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.322175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.322225 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.322308 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:26.322569 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.323229 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.324421 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.324896 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.325013 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.325078 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.325878 (+ 800us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.325948 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:25:26.326112 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21eb6358f3dd43a3abf56eb3f597f9c8" candidate_uuid: "ef7a5385aae84f3192e11aa9e82c233a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c08cb6a0209647a49169edae08439ffc" is_pre_election: true
I20260926 07:25:26.326435 7713 raft_consensus.cc:2454] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef7a5385aae84f3192e11aa9e82c233a in term 0.
I20260926 07:25:26.326794 7439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21eb6358f3dd43a3abf56eb3f597f9c8" candidate_uuid: "ef7a5385aae84f3192e11aa9e82c233a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" is_pre_election: true
I20260926 07:25:26.326924 7510 leader_election.cc:302] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c08cb6a0209647a49169edae08439ffc, ef7a5385aae84f3192e11aa9e82c233a; no voters:
I20260926 07:25:26.327148 7439 raft_consensus.cc:2454] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef7a5385aae84f3192e11aa9e82c233a in term 0.
I20260926 07:25:26.327317 7800 raft_consensus.cc:2781] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:26.327421 7800 raft_consensus.cc:482] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:26.327467 7800 raft_consensus.cc:3037] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:26.328490 7800 raft_consensus.cc:504] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } }
I20260926 07:25:26.328841 7800 leader_election.cc:288] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [CANDIDATE]: Term 1 election: Requested vote from peers 852fb4b9b0fb4ed187c9f283d7bc3132 (127.0.98.1:37491), c08cb6a0209647a49169edae08439ffc (127.0.98.3:34843)
I20260926 07:25:26.329157 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21eb6358f3dd43a3abf56eb3f597f9c8" candidate_uuid: "ef7a5385aae84f3192e11aa9e82c233a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c08cb6a0209647a49169edae08439ffc"
I20260926 07:25:26.329155 7439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21eb6358f3dd43a3abf56eb3f597f9c8" candidate_uuid: "ef7a5385aae84f3192e11aa9e82c233a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132"
I20260926 07:25:26.329288 7713 raft_consensus.cc:3037] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:26.329312 7439 raft_consensus.cc:3037] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:26.330255 7439 raft_consensus.cc:2454] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef7a5385aae84f3192e11aa9e82c233a in term 1.
I20260926 07:25:26.330300 7713 raft_consensus.cc:2454] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef7a5385aae84f3192e11aa9e82c233a in term 1.
I20260926 07:25:26.330536 7512 leader_election.cc:302] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852fb4b9b0fb4ed187c9f283d7bc3132, ef7a5385aae84f3192e11aa9e82c233a; no voters:
I20260926 07:25:26.330737 7800 raft_consensus.cc:2781] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:26.330960 7800 raft_consensus.cc:686] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [term 1 LEADER]: Becoming Leader. State: Replica: ef7a5385aae84f3192e11aa9e82c233a, State: Running, Role: LEADER
I20260926 07:25:26.331323 7800 consensus_queue.cc:237] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } }
I20260926 07:25:26.335162 6522 catalog_manager.cc:5495] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a reported cstate change: term changed from 0 to 1, leader changed from <none> to ef7a5385aae84f3192e11aa9e82c233a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ef7a5385aae84f3192e11aa9e82c233a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } health_report { overall_health: UNKNOWN } } }
I20260926 07:25:26.371109 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.367390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.0.98.2:36701)
0926 07:25:26.367533 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.367538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.367552 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:26.367789 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.367792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.367851 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.367965 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:26.368208 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.368730 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.369589 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.369865 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.369977 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.370291 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:26.370710 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:26.370849 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.370886 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.370944 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:25:26.377022 7439 raft_consensus.cc:1261] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 1 FOLLOWER]: Refusing update from remote peer ef7a5385aae84f3192e11aa9e82c233a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:26.377048 7713 raft_consensus.cc:1261] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc [term 1 FOLLOWER]: Refusing update from remote peer ef7a5385aae84f3192e11aa9e82c233a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:26.377780 7800 consensus_queue.cc:1035] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [LEADER]: Connected to new peer: Peer: permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:26.377986 7802 consensus_queue.cc:1035] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:25:26.383131 7762 tablet.cc:2369] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:25:26.387195 7807 mvcc.cc:205] Tried to move back new op lower bound from 7333509228033372160 to 7333509227855761408. Current Snapshot: MvccSnapshot[applied={T|T < 7333509228033372160}]
I20260926 07:25:26.387017 7806 mvcc.cc:205] Tried to move back new op lower bound from 7333509228033372160 to 7333509227855761408. Current Snapshot: MvccSnapshot[applied={T|T < 7333509228033372160}]
I20260926 07:25:26.396281 6522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:26.396451 6522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:26.400503 6522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46dc84ef76be4b5181e29f5cd68aacd5: Sending DeleteTablet for 3 replicas of tablet 21eb6358f3dd43a3abf56eb3f597f9c8
I20260926 07:25:26.401389 7693 tablet_service.cc:1514] Processing DeleteTablet for tablet 21eb6358f3dd43a3abf56eb3f597f9c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44128
I20260926 07:25:26.401376 7556 tablet_service.cc:1514] Processing DeleteTablet for tablet 21eb6358f3dd43a3abf56eb3f597f9c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43946
I20260926 07:25:26.401895 7816 tablet_replica.cc:330] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc: stopping tablet replica
I20260926 07:25:26.402209 7419 tablet_service.cc:1514] Processing DeleteTablet for tablet 21eb6358f3dd43a3abf56eb3f597f9c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:26 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54058
I20260926 07:25:26.402423 7815 tablet_replica.cc:330] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a: stopping tablet replica
I20260926 07:25:26.402782 7815 raft_consensus.cc:2229] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:25:26.402969 7816 raft_consensus.cc:2229] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:26.403208 7815 raft_consensus.cc:2258] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:26.403568 7816 raft_consensus.cc:2258] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:26.404093 7817 tablet_replica.cc:330] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132: stopping tablet replica
I20260926 07:25:26.404363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 6490
I20260926 07:25:26.405154 7816 ts_tablet_manager.cc:1905] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:26.405943 7815 ts_tablet_manager.cc:1905] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:26.408140 7816 ts_tablet_manager.cc:1918] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:26.408226 7816 log.cc:1196] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/21eb6358f3dd43a3abf56eb3f597f9c8
I20260926 07:25:26.408537 7816 ts_tablet_manager.cc:1939] T 21eb6358f3dd43a3abf56eb3f597f9c8 P c08cb6a0209647a49169edae08439ffc: Deleting consensus metadata
I20260926 07:25:26.408735 7815 ts_tablet_manager.cc:1918] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:26.408824 7815 log.cc:1196] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/21eb6358f3dd43a3abf56eb3f597f9c8
I20260926 07:25:26.409124 7815 ts_tablet_manager.cc:1939] T 21eb6358f3dd43a3abf56eb3f597f9c8 P ef7a5385aae84f3192e11aa9e82c233a: Deleting consensus metadata
I20260926 07:25:26.411559 7817 raft_consensus.cc:2229] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:26.411964 7817 raft_consensus.cc:2258] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:26.413630 7817 ts_tablet_manager.cc:1905] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:25:26.413895 7646 connection.cc:535] server connection from 127.0.0.1:44128 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 07:25:26.414279 7512 connection.cc:535] server connection from 127.0.0.1:43946 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:25:26.416781 7817 ts_tablet_manager.cc:1918] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:26.416958 7817 log.cc:1196] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-0/wal/wals/21eb6358f3dd43a3abf56eb3f597f9c8
I20260926 07:25:26.417371 7817 ts_tablet_manager.cc:1939] T 21eb6358f3dd43a3abf56eb3f597f9c8 P 852fb4b9b0fb4ed187c9f283d7bc3132: Deleting consensus metadata
W20260926 07:25:26.418485 7375 connection.cc:441] server connection from 127.0.0.1:54058 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54058 (request call id 1) could send its response
I20260926 07:25:26.420001 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.403810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.0.98.61:38947)
0926 07:25:26.404020 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.404026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.410435 (+ 6409us) server_negotiation.cc:407] Connection header received
0926 07:25:26.410565 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.410569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.410622 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.410710 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:26.411731 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.412545 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.413756 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.414087 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.414159 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.416374 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:26.416378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:26.416381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:26.416408 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:26.418087 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.418579 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.418583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.418585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.418663 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.419088 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.419091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.419093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.419307 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:26.419420 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.419732 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.419844 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260926 07:25:26.428061 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.418890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.0.98.60:46361)
0926 07:25:26.419024 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.419047 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.419226 (+ 179us) server_negotiation.cc:407] Connection header received
0926 07:25:26.419462 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.419466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.419518 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.419613 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:26.420335 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.421020 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.422148 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.422395 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.422509 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.424132 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:26.424136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:26.424138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:26.424160 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:26.425975 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.426450 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.426454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.426456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.426540 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.426864 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.426867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.426869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.427141 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:26.427457 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.427772 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.427893 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:25:26.429455 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.403715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.0.98.60:46361)
0926 07:25:26.403928 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.403933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.420513 (+ 16580us) server_negotiation.cc:407] Connection header received
0926 07:25:26.420593 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.420596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.420637 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.420725 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:26.421430 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.422188 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.423333 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.423743 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.423855 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.425490 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:26.425496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:26.425500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:26.425527 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:26.427448 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.427916 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.427919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.427921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.428013 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.428570 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.428573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.428575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.428812 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:26.428929 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.429265 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.429361 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260926 07:25:26.436491 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:26.427560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.0.98.61:38947)
0926 07:25:26.427681 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:26.427685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:26.428014 (+ 329us) server_negotiation.cc:407] Connection header received
0926 07:25:26.428212 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:26.428218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:26.428261 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:26.428307 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:26.429250 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.430025 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.430989 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:26.431353 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:26.431456 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:26.433175 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:26.433179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:26.433181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:26.433204 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:26.434918 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.435418 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.435420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.435422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.435496 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:26.435752 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:26.435754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:26.435755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:26.435918 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:26.436016 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:26.436233 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 07:25:26.436327 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:25:27.391552 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.390563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.3:40371)
0926 07:25:27.391296 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:27.391398 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":98,"threads_started":1}
I20260926 07:25:27.391877 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.391249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.2:58833)
0926 07:25:27.391507 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:27.391766 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":61,"threads_started":1}
W20260926 07:25:27.392127 7759 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33023 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
W20260926 07:25:27.392129 7622 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33023 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
I20260926 07:25:27.392354 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.391892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.1:54473)
0926 07:25:27.392172 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:27.392260 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":58,"threads_started":1}
W20260926 07:25:27.392628 7485 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33023 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
I20260926 07:25:27.900574 7828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46dc84ef76be4b5181e29f5cd68aacd5)
I20260926 07:25:27.900599 7829 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46dc84ef76be4b5181e29f5cd68aacd5)
I20260926 07:25:27.900835 7829 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } }
I20260926 07:25:27.900866 7828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } }
I20260926 07:25:27.901856 7828 leader_election.cc:288] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46dc84ef76be4b5181e29f5cd68aacd5 (127.0.98.62:33023), fd9da486a92542fcb2304179a2ab9f4c (127.0.98.61:38947)
I20260926 07:25:27.902166 7829 leader_election.cc:288] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46dc84ef76be4b5181e29f5cd68aacd5 (127.0.98.62:33023), 10b3febc784444d3967e52d3992e7c40 (127.0.98.60:46361)
I20260926 07:25:27.902616 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.901976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:52924)
0926 07:25:27.902420 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:27.902501 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":79,"threads_started":1}
I20260926 07:25:27.903131 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.901974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:52926)
0926 07:25:27.902834 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:27.902976 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":183,"thread_start_us":53,"threads_started":1}
W20260926 07:25:27.903338 7110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
W20260926 07:25:27.903795 7292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
I20260926 07:25:27.904485 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.904261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:52954)
0926 07:25:27.904371 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:27.904422 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:25:27.904599 7110 leader_election.cc:334] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 46dc84ef76be4b5181e29f5cd68aacd5 (127.0.98.62:33023): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
I20260926 07:25:27.904906 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.904630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:52962)
0926 07:25:27.904754 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:27.904808 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 07:25:27.905019 7292 leader_election.cc:334] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 46dc84ef76be4b5181e29f5cd68aacd5 (127.0.98.62:33023): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
I20260926 07:25:27.907444 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.902199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38947 (local address 127.0.0.1:58526)
0926 07:25:27.902769 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:27.902784 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:27.902871 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:27.903509 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:27.903513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:27.903536 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:27.903669 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:27.903677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:27.904866 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:27.904870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:27.906061 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:27.906078 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:27.906951 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:27.906955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:27.907123 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:27.907143 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:25:27.907235 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:25:27.907304 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":51,"thread_start_us":57,"threads_started":1}
I20260926 07:25:27.907460 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.901974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:54890)
0926 07:25:27.902393 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:27.902409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:27.902516 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:27.903614 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:27.903617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:27.903632 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:27.903720 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:27.903728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:27.904608 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:27.904611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:27.905911 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:27.905921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:27.906951 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:27.906955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:27.907117 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:27.907143 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:25:27.907235 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:25:27.907304 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":20,"thread_start_us":58,"threads_started":1}
I20260926 07:25:27.907487 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.902451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.0.98.61:38947)
0926 07:25:27.902940 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:27.902946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:27.902964 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:25:27.903023 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:27.903053 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:27.903121 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:27.903324 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:27.903803 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:27.904752 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:27.906191 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:27.906835 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:27.906942 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:27.907104 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:27.907267 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 07:25:27.907345 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":407,"thread_start_us":58,"threads_started":1}
I20260926 07:25:27.907485 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.903106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.0.98.60:46361)
0926 07:25:27.903343 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:27.903348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:27.903360 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:25:27.903402 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:27.903406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:27.903453 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:27.903598 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:27.903850 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:27.904481 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:27.906051 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:27.906802 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:27.906935 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:27.907095 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:27.907267 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 07:25:27.907345 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":171,"thread_start_us":546,"threads_started":1}
I20260926 07:25:27.907972 7133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10b3febc784444d3967e52d3992e7c40" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd9da486a92542fcb2304179a2ab9f4c" is_pre_election: true
I20260926 07:25:27.908108 7311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd9da486a92542fcb2304179a2ab9f4c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "10b3febc784444d3967e52d3992e7c40" is_pre_election: true
I20260926 07:25:27.908213 7133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b3febc784444d3967e52d3992e7c40 in term 1.
I20260926 07:25:27.908253 7311 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd9da486a92542fcb2304179a2ab9f4c in term 1.
I20260926 07:25:27.908550 7290 leader_election.cc:302] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10b3febc784444d3967e52d3992e7c40, fd9da486a92542fcb2304179a2ab9f4c; no voters: 46dc84ef76be4b5181e29f5cd68aacd5
I20260926 07:25:27.908584 7109 leader_election.cc:302] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10b3febc784444d3967e52d3992e7c40, fd9da486a92542fcb2304179a2ab9f4c; no voters: 46dc84ef76be4b5181e29f5cd68aacd5
I20260926 07:25:27.908766 7828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 07:25:27.908835 7829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 07:25:27.908852 7828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46dc84ef76be4b5181e29f5cd68aacd5)
I20260926 07:25:27.908900 7829 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46dc84ef76be4b5181e29f5cd68aacd5)
I20260926 07:25:27.908932 7828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:25:27.908951 7829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:25:27.909920 7829 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } }
I20260926 07:25:27.909926 7828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } }
I20260926 07:25:27.910282 7828 leader_election.cc:288] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [CANDIDATE]: Term 2 election: Requested vote from peers 46dc84ef76be4b5181e29f5cd68aacd5 (127.0.98.62:33023), fd9da486a92542fcb2304179a2ab9f4c (127.0.98.61:38947)
I20260926 07:25:27.910300 7829 leader_election.cc:288] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [CANDIDATE]: Term 2 election: Requested vote from peers 46dc84ef76be4b5181e29f5cd68aacd5 (127.0.98.62:33023), 10b3febc784444d3967e52d3992e7c40 (127.0.98.60:46361)
I20260926 07:25:27.910591 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.910358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:52968)
0926 07:25:27.910445 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:27.910498 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 07:25:27.910780 7133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10b3febc784444d3967e52d3992e7c40" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd9da486a92542fcb2304179a2ab9f4c"
I20260926 07:25:27.910849 7311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd9da486a92542fcb2304179a2ab9f4c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "10b3febc784444d3967e52d3992e7c40"
I20260926 07:25:27.910888 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.910379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:52974)
0926 07:25:27.910721 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:27.910772 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260926 07:25:27.911007 7311 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fd9da486a92542fcb2304179a2ab9f4c in current term 2: Already voted for candidate 10b3febc784444d3967e52d3992e7c40 in this term.
I20260926 07:25:27.911010 7133 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 10b3febc784444d3967e52d3992e7c40 in current term 2: Already voted for candidate fd9da486a92542fcb2304179a2ab9f4c in this term.
I20260926 07:25:27.911535 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.911278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:52984)
0926 07:25:27.911408 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:27.911462 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:25:27.911656 7292 leader_election.cc:334] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 46dc84ef76be4b5181e29f5cd68aacd5 (127.0.98.62:33023): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
I20260926 07:25:27.911760 7292 leader_election.cc:302] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10b3febc784444d3967e52d3992e7c40; no voters: 46dc84ef76be4b5181e29f5cd68aacd5, fd9da486a92542fcb2304179a2ab9f4c
I20260926 07:25:27.911912 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:27.911679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:52998)
0926 07:25:27.911786 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:27.911844 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 07:25:27.911912 7828 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260926 07:25:27.912039 7110 leader_election.cc:334] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 46dc84ef76be4b5181e29f5cd68aacd5 (127.0.98.62:33023): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
I20260926 07:25:27.912118 7110 leader_election.cc:302] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd9da486a92542fcb2304179a2ab9f4c; no voters: 10b3febc784444d3967e52d3992e7c40, 46dc84ef76be4b5181e29f5cd68aacd5
I20260926 07:25:27.912238 7829 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 07:25:28.393388 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:28.392634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.3:42301)
0926 07:25:28.393202 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:28.393272 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":80,"threads_started":1}
I20260926 07:25:28.393895 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:28.393148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.1:42669)
0926 07:25:28.393755 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:28.393811 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":69,"threads_started":1}
I20260926 07:25:28.393895 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:28.392602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.2:54489)
0926 07:25:28.393766 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:28.393814 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1101,"thread_start_us":94,"threads_started":1}
I20260926 07:25:29.394801 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:29.394078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.3:36841)
0926 07:25:29.394589 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:29.394680 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":82,"threads_started":1}
I20260926 07:25:29.395532 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:29.394708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.1:56609)
0926 07:25:29.395384 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:29.395441 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":67,"threads_started":1}
I20260926 07:25:29.395877 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:29.394772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.2:50403)
0926 07:25:29.395741 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:29.395807 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":70,"threads_started":1}
I20260926 07:25:30.101651 7844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:30.101783 7844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } }
I20260926 07:25:30.102232 7844 leader_election.cc:288] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 46dc84ef76be4b5181e29f5cd68aacd5 (127.0.98.62:33023), fd9da486a92542fcb2304179a2ab9f4c (127.0.98.61:38947)
I20260926 07:25:30.102766 7133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10b3febc784444d3967e52d3992e7c40" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd9da486a92542fcb2304179a2ab9f4c" is_pre_election: true
I20260926 07:25:30.102813 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.102393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:53258)
0926 07:25:30.102635 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.102723 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
I20260926 07:25:30.102890 7133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b3febc784444d3967e52d3992e7c40 in term 2.
I20260926 07:25:30.103255 7290 leader_election.cc:302] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b3febc784444d3967e52d3992e7c40, fd9da486a92542fcb2304179a2ab9f4c; no voters:
I20260926 07:25:30.103417 7844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 07:25:30.103472 7844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:30.103523 7844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:25:30.103922 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.103673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:53260)
0926 07:25:30.103795 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.103855 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:25:30.104060 7292 leader_election.cc:334] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 46dc84ef76be4b5181e29f5cd68aacd5 (127.0.98.62:33023): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
I20260926 07:25:30.104691 7844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } }
I20260926 07:25:30.105060 7844 leader_election.cc:288] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [CANDIDATE]: Term 3 election: Requested vote from peers 46dc84ef76be4b5181e29f5cd68aacd5 (127.0.98.62:33023), fd9da486a92542fcb2304179a2ab9f4c (127.0.98.61:38947)
I20260926 07:25:30.105299 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.105116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:53264)
0926 07:25:30.105208 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.105257 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 07:25:30.105441 7133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10b3febc784444d3967e52d3992e7c40" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd9da486a92542fcb2304179a2ab9f4c"
I20260926 07:25:30.105588 7133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:25:30.106004 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.105788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:53276)
0926 07:25:30.105893 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.105940 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 07:25:30.106122 7292 leader_election.cc:334] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 46dc84ef76be4b5181e29f5cd68aacd5 (127.0.98.62:33023): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
I20260926 07:25:30.106613 7133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b3febc784444d3967e52d3992e7c40 in term 3.
I20260926 07:25:30.106892 7290 leader_election.cc:302] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10b3febc784444d3967e52d3992e7c40, fd9da486a92542fcb2304179a2ab9f4c; no voters: 46dc84ef76be4b5181e29f5cd68aacd5
I20260926 07:25:30.107120 7844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:25:30.107309 7844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [term 3 LEADER]: Becoming Leader. State: Replica: 10b3febc784444d3967e52d3992e7c40, State: Running, Role: LEADER
I20260926 07:25:30.107561 7844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } }
I20260926 07:25:30.108734 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10b3febc784444d3967e52d3992e7c40. Latest consensus state: current_term: 3 leader_uuid: "10b3febc784444d3967e52d3992e7c40" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } } }
I20260926 07:25:30.108824 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:30.109086 7849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:30.111505 7849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:30.111932 7849 catalog_manager.cc:1259] Loaded cluster ID: 28dc622c9eb64edebd3dde2b34769ebe
I20260926 07:25:30.111980 7849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:30.112565 7849 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:30.112987 7849 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: Loaded TSK: 0
I20260926 07:25:30.113669 7849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:25:30.137830 7300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:25:30.140969 7300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:30.144214 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.143929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:53298)
0926 07:25:30.144069 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.144131 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 07:25:30.144243 7133 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [term 3 FOLLOWER]: Refusing update from remote peer 10b3febc784444d3967e52d3992e7c40: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 07:25:30.144711 7847 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 07:25:30.144939 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.144714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.0.1:53314)
0926 07:25:30.144811 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.144880 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 07:25:30.145056 7292 consensus_peers.cc:487] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 -> Peer 46dc84ef76be4b5181e29f5cd68aacd5 (127.0.98.62:33023): Couldn't send request to peer 46dc84ef76be4b5181e29f5cd68aacd5. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 07:25:30.146204 7851 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10b3febc784444d3967e52d3992e7c40. Latest consensus state: current_term: 3 leader_uuid: "10b3febc784444d3967e52d3992e7c40" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } } }
I20260926 07:25:30.146348 7851 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:25:30.147578 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "10b3febc784444d3967e52d3992e7c40" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } } }
I20260926 07:25:30.147727 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:30.147970 7851 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "10b3febc784444d3967e52d3992e7c40" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "46dc84ef76be4b5181e29f5cd68aacd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33023 } } peers { permanent_uuid: "fd9da486a92542fcb2304179a2ab9f4c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38947 } attrs { promote: false } } peers { permanent_uuid: "10b3febc784444d3967e52d3992e7c40" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attrs { promote: false } } }
I20260926 07:25:30.148097 7851 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9da486a92542fcb2304179a2ab9f4c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:25:30.158155 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.153826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34843 (local address 127.0.0.1:54632)
0926 07:25:30.154598 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.154610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:30.154679 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:30.155150 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:30.155153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:30.155169 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:30.155246 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:30.155252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.156115 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.156121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:30.156873 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:30.156881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.157743 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.157746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:30.157837 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:30.157860 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:25:30.157929 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:25:30.157991 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":310,"thread_start_us":85,"threads_started":1}
I20260926 07:25:30.158351 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.153686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.0.98.1:37491)
0926 07:25:30.154065 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:30.154070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:30.154085 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:25:30.154147 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:30.154153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:30.154211 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:30.154600 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:30.154721 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.155503 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.157085 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.157754 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.157912 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:30.157975 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:30.158111 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 07:25:30.158212 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":80,"threads_started":1}
I20260926 07:25:30.158273 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.153561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37491 (local address 127.0.0.1:55202)
0926 07:25:30.153938 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.153955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:30.154051 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:30.154351 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:30.154356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:30.154377 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:30.154493 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:30.154500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.155644 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.155647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:30.156935 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:30.156945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.157900 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.157906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:30.157991 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:30.158008 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 07:25:30.158076 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 07:25:30.158127 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
I20260926 07:25:30.158752 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.154006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54632 (local address 127.0.98.3:34843)
0926 07:25:30.154880 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:30.154885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:30.154898 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:25:30.154947 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:30.154950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:30.154992 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:30.155340 (+ 348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:30.155432 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.156009 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.156995 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.157618 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.157747 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:30.157820 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:30.158505 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 07:25:30.158587 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":80,"threads_started":1}
I20260926 07:25:30.159240 7419 tablet_service.cc:1467] Processing CreateTablet for tablet 4e145dde20c941cbb245b82cd2f21c6d (DEFAULT_TABLE table=test-table [id=a2a154d0f6034681a4beeda4e8b94ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:30.159323 7693 tablet_service.cc:1467] Processing CreateTablet for tablet 4e145dde20c941cbb245b82cd2f21c6d (DEFAULT_TABLE table=test-table [id=a2a154d0f6034681a4beeda4e8b94ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:30.159540 7693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e145dde20c941cbb245b82cd2f21c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:30.159534 7419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e145dde20c941cbb245b82cd2f21c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:30.162189 7863 tablet_bootstrap.cc:492] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc: Bootstrap starting.
I20260926 07:25:30.162618 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.154381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.0.98.2:36701)
0926 07:25:30.154755 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:30.154760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:30.155027 (+ 267us) server_negotiation.cc:407] Connection header received
0926 07:25:30.155185 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:30.155189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:30.155231 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:30.155331 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:30.155578 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.156442 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.158256 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.158946 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.159111 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:30.159179 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:30.162401 (+ 3222us) server_negotiation.cc:299] Negotiation successful
0926 07:25:30.162487 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":81,"threads_started":1}
I20260926 07:25:30.162941 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.154258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36701 (local address 127.0.0.1:51724)
0926 07:25:30.154935 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.154947 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:30.155082 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:30.155339 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:30.155342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:30.155358 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:30.155455 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:30.155460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.156579 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.156583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:30.158116 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:30.158126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.159095 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.162114 (+ 3019us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:30.162238 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:30.162263 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:25:30.162769 (+ 506us) client_negotiation.cc:240] Negotiation successful
0926 07:25:30.162850 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":92,"threads_started":1}
I20260926 07:25:30.163326 7863 tablet_bootstrap.cc:654] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:30.163540 7556 tablet_service.cc:1467] Processing CreateTablet for tablet 4e145dde20c941cbb245b82cd2f21c6d (DEFAULT_TABLE table=test-table [id=a2a154d0f6034681a4beeda4e8b94ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:30.163822 7556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e145dde20c941cbb245b82cd2f21c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:30.163985 7862 tablet_bootstrap.cc:492] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132: Bootstrap starting.
I20260926 07:25:30.164528 7863 tablet_bootstrap.cc:492] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc: No bootstrap required, opened a new log
I20260926 07:25:30.164662 7863 ts_tablet_manager.cc:1397] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:25:30.165051 7862 tablet_bootstrap.cc:654] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:30.165143 7863 raft_consensus.cc:348] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } }
I20260926 07:25:30.165257 7863 raft_consensus.cc:374] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:30.165341 7863 raft_consensus.cc:729] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08cb6a0209647a49169edae08439ffc, State: Initialized, Role: FOLLOWER
I20260926 07:25:30.165534 7863 consensus_queue.cc:260] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } }
I20260926 07:25:30.165920 7863 ts_tablet_manager.cc:1428] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:30.166219 7862 tablet_bootstrap.cc:492] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132: No bootstrap required, opened a new log
I20260926 07:25:30.166298 7862 ts_tablet_manager.cc:1397] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:30.166711 7862 raft_consensus.cc:348] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } }
I20260926 07:25:30.166826 7862 raft_consensus.cc:374] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:30.166858 7862 raft_consensus.cc:729] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852fb4b9b0fb4ed187c9f283d7bc3132, State: Initialized, Role: FOLLOWER
I20260926 07:25:30.167161 7867 tablet_bootstrap.cc:492] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a: Bootstrap starting.
I20260926 07:25:30.166972 7862 consensus_queue.cc:260] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } }
I20260926 07:25:30.167431 7862 ts_tablet_manager.cc:1428] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:30.168373 7867 tablet_bootstrap.cc:654] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:30.169130 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.166365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.3:39719)
0926 07:25:30.168976 (+ 2611us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.169038 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2537,"thread_start_us":94,"threads_started":1}
I20260926 07:25:30.169394 7760 heartbeater.cc:503] Master 127.0.98.60:46361 was elected leader, sending a full tablet report...
I20260926 07:25:30.170174 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.168574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.1:56609)
0926 07:25:30.170011 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.170085 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1373,"thread_start_us":87,"threads_started":1}
I20260926 07:25:30.170562 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.169607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.3:40239)
0926 07:25:30.169818 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.169875 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20260926 07:25:30.170993 7486 heartbeater.cc:503] Master 127.0.98.60:46361 was elected leader, sending a full tablet report...
I20260926 07:25:30.171010 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.170723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.1:42241)
0926 07:25:30.170866 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.170927 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 07:25:30.171519 7867 tablet_bootstrap.cc:492] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a: No bootstrap required, opened a new log
I20260926 07:25:30.171608 7867 ts_tablet_manager.cc:1397] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 07:25:30.172021 7867 raft_consensus.cc:348] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } }
I20260926 07:25:30.172154 7867 raft_consensus.cc:374] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:30.172187 7867 raft_consensus.cc:729] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef7a5385aae84f3192e11aa9e82c233a, State: Initialized, Role: FOLLOWER
I20260926 07:25:30.172327 7867 consensus_queue.cc:260] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } }
I20260926 07:25:30.172669 7867 ts_tablet_manager.cc:1428] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:30.173424 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.172918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.2:57925)
0926 07:25:30.173269 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.173324 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":84,"threads_started":1}
I20260926 07:25:30.174053 7623 heartbeater.cc:503] Master 127.0.98.60:46361 was elected leader, sending a full tablet report...
I20260926 07:25:30.174216 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.173807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.2:43245)
0926 07:25:30.174063 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.174141 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":83,"threads_started":1}
W20260926 07:25:30.185108 7625 tablet.cc:2369] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:25:30.202721 7488 tablet.cc:2369] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:25:30.435127 7869 raft_consensus.cc:482] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:30.435292 7869 raft_consensus.cc:504] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } }
I20260926 07:25:30.436599 7869 leader_election.cc:288] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c08cb6a0209647a49169edae08439ffc (127.0.98.3:34843), ef7a5385aae84f3192e11aa9e82c233a (127.0.98.2:36701)
I20260926 07:25:30.440411 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.436791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34843 (local address 127.0.98.1:51589)
0926 07:25:30.436928 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.436949 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:30.437060 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:30.437401 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:30.437404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:30.437425 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:30.437534 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:30.437538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.438259 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.438266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:30.439199 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:30.439208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.439877 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.439881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:30.440005 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:30.440091 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 07:25:30.440172 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:25:30.440261 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":85}
I20260926 07:25:30.440410 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.436878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51589 (local address 127.0.98.3:34843)
0926 07:25:30.437023 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:30.437028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:30.437047 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:25:30.437209 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:30.437212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:30.437256 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:30.437347 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:30.437645 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.438148 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.439319 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.439723 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.439857 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:30.439929 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:30.440203 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 07:25:30.440294 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:25:30.440485 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.437076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49623 (local address 127.0.98.2:36701)
0926 07:25:30.437190 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:30.437194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:30.437324 (+ 130us) server_negotiation.cc:407] Connection header received
0926 07:25:30.437481 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:30.437484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:30.437536 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:30.437620 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:30.437872 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.438355 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.439445 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.439827 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.439942 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:30.440006 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:30.440290 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 07:25:30.440368 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:25:30.440811 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.436891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36701 (local address 127.0.98.1:49623)
0926 07:25:30.437219 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.437232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:30.437365 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:30.437631 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:30.437634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:30.437648 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:30.437770 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:30.437775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.438469 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.438474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:30.439327 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:30.439335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.439998 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.440001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:30.440152 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:30.440178 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:25:30.440628 (+ 450us) client_negotiation.cc:240] Negotiation successful
0926 07:25:30.440695 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":23,"thread_start_us":129,"threads_started":1}
I20260926 07:25:30.441365 7576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e145dde20c941cbb245b82cd2f21c6d" candidate_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef7a5385aae84f3192e11aa9e82c233a" is_pre_election: true
I20260926 07:25:30.441447 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e145dde20c941cbb245b82cd2f21c6d" candidate_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c08cb6a0209647a49169edae08439ffc" is_pre_election: true
I20260926 07:25:30.441604 7713 raft_consensus.cc:2454] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 852fb4b9b0fb4ed187c9f283d7bc3132 in term 0.
I20260926 07:25:30.441613 7576 raft_consensus.cc:2454] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 852fb4b9b0fb4ed187c9f283d7bc3132 in term 0.
I20260926 07:25:30.441993 7373 leader_election.cc:302] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852fb4b9b0fb4ed187c9f283d7bc3132, c08cb6a0209647a49169edae08439ffc; no voters:
I20260926 07:25:30.442322 7869 raft_consensus.cc:2781] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:30.442405 7869 raft_consensus.cc:482] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:30.442453 7869 raft_consensus.cc:3037] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:30.443567 7869 raft_consensus.cc:504] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } }
I20260926 07:25:30.443929 7869 leader_election.cc:288] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [CANDIDATE]: Term 1 election: Requested vote from peers c08cb6a0209647a49169edae08439ffc (127.0.98.3:34843), ef7a5385aae84f3192e11aa9e82c233a (127.0.98.2:36701)
I20260926 07:25:30.444195 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e145dde20c941cbb245b82cd2f21c6d" candidate_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c08cb6a0209647a49169edae08439ffc"
I20260926 07:25:30.444319 7713 raft_consensus.cc:3037] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:30.444259 7576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e145dde20c941cbb245b82cd2f21c6d" candidate_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef7a5385aae84f3192e11aa9e82c233a"
I20260926 07:25:30.444414 7576 raft_consensus.cc:3037] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:30.445334 7713 raft_consensus.cc:2454] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 852fb4b9b0fb4ed187c9f283d7bc3132 in term 1.
I20260926 07:25:30.445354 7576 raft_consensus.cc:2454] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 852fb4b9b0fb4ed187c9f283d7bc3132 in term 1.
I20260926 07:25:30.445595 7373 leader_election.cc:302] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852fb4b9b0fb4ed187c9f283d7bc3132, c08cb6a0209647a49169edae08439ffc; no voters:
I20260926 07:25:30.445775 7869 raft_consensus.cc:2781] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:30.445960 7869 raft_consensus.cc:686] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [term 1 LEADER]: Becoming Leader. State: Replica: 852fb4b9b0fb4ed187c9f283d7bc3132, State: Running, Role: LEADER
I20260926 07:25:30.446192 7869 consensus_queue.cc:237] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } }
I20260926 07:25:30.446607 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.446403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.1:54833)
0926 07:25:30.446497 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.446545 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 07:25:30.448513 7306 catalog_manager.cc:5495] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 reported cstate change: term changed from 0 to 1, leader changed from <none> to 852fb4b9b0fb4ed187c9f283d7bc3132 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852fb4b9b0fb4ed187c9f283d7bc3132" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 } health_report { overall_health: UNKNOWN } } }
I20260926 07:25:30.519997 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.516384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.0.98.1:37491)
0926 07:25:30.516516 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:30.516522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:30.516536 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:30.516716 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:30.516718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:30.516775 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:30.516889 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:30.517166 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.517703 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.518545 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:30.518762 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:30.518871 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:30.519158 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:30.519600 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:30.519730 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:30.519793 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 07:25:30.519849 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:25:30.522607 7713 raft_consensus.cc:1261] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc [term 1 FOLLOWER]: Refusing update from remote peer 852fb4b9b0fb4ed187c9f283d7bc3132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:30.522912 7576 raft_consensus.cc:1261] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a [term 1 FOLLOWER]: Refusing update from remote peer 852fb4b9b0fb4ed187c9f283d7bc3132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:30.523157 7877 consensus_queue.cc:1035] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c08cb6a0209647a49169edae08439ffc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:30.523506 7869 consensus_queue.cc:1035] T 4e145dde20c941cbb245b82cd2f21c6d P 852fb4b9b0fb4ed187c9f283d7bc3132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef7a5385aae84f3192e11aa9e82c233a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:30.525182 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.524910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.2:40259)
0926 07:25:30.525055 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.525112 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 07:25:30.526369 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.525229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.3:42185)
0926 07:25:30.526241 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.526298 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":925}
I20260926 07:25:30.528004 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.527767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.1:36103)
0926 07:25:30.527874 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.527928 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 07:25:30.528554 7883 mvcc.cc:205] Tried to move back new op lower bound from 7333509245015863296 to 7333509244710162432. Current Snapshot: MvccSnapshot[applied={T|T < 7333509245015863296}]
I20260926 07:25:30.528769 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.528514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.2:40455)
0926 07:25:30.528657 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.528709 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 07:25:30.532510 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.532208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.3:38743)
0926 07:25:30.532388 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.532437 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 07:25:30.532680 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.532435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33023 (local address 127.0.98.1:49113)
0926 07:25:30.532574 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.532621 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 07:25:30.537829 7300 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:30.537992 7300 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:30.541466 7300 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10b3febc784444d3967e52d3992e7c40: Sending DeleteTablet for 3 replicas of tablet 4e145dde20c941cbb245b82cd2f21c6d
I20260926 07:25:30.542223 7556 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e145dde20c941cbb245b82cd2f21c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:30 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:51724
I20260926 07:25:30.542547 7693 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e145dde20c941cbb245b82cd2f21c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:30 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54632
I20260926 07:25:30.542557 7419 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e145dde20c941cbb245b82cd2f21c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:30 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55202
I20260926 07:25:30.542615 7890 tablet_replica.cc:330] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a: stopping tablet replica
I20260926 07:25:30.542745 7890 raft_consensus.cc:2229] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:30.542891 7892 tablet_replica.cc:330] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc: stopping tablet replica
I20260926 07:25:30.542903 7890 raft_consensus.cc:2258] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:30.543023 7892 raft_consensus.cc:2229] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:30.543483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 7268
I20260926 07:25:30.544421 7890 ts_tablet_manager.cc:1905] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:30.545562 7892 raft_consensus.cc:2258] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:30.546250 7892 ts_tablet_manager.cc:1905] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:30.547281 7890 ts_tablet_manager.cc:1918] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:30.547399 7890 log.cc:1196] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/4e145dde20c941cbb245b82cd2f21c6d
I20260926 07:25:30.547732 7890 ts_tablet_manager.cc:1939] T 4e145dde20c941cbb245b82cd2f21c6d P ef7a5385aae84f3192e11aa9e82c233a: Deleting consensus metadata
I20260926 07:25:30.548444 7290 catalog_manager.cc:4841] TS ef7a5385aae84f3192e11aa9e82c233a (127.0.98.2:36701): tablet 4e145dde20c941cbb245b82cd2f21c6d (table test-table [id=a2a154d0f6034681a4beeda4e8b94ca0]) successfully deleted
I20260926 07:25:30.548679 7892 ts_tablet_manager.cc:1918] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:30.548776 7892 log.cc:1196] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/4e145dde20c941cbb245b82cd2f21c6d
I20260926 07:25:30.549015 7892 ts_tablet_manager.cc:1939] T 4e145dde20c941cbb245b82cd2f21c6d P c08cb6a0209647a49169edae08439ffc: Deleting consensus metadata
I20260926 07:25:30.549592 7291 catalog_manager.cc:4841] TS c08cb6a0209647a49169edae08439ffc (127.0.98.3:34843): tablet 4e145dde20c941cbb245b82cd2f21c6d (table test-table [id=a2a154d0f6034681a4beeda4e8b94ca0]) successfully deleted
I20260926 07:25:30.551961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 7493
I20260926 07:25:30.552304 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:30.552006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37491 (local address 127.0.0.1:55216)
0926 07:25:30.552157 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:30.552216 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:25:30.553637 7293 catalog_manager.cc:4584] TS 852fb4b9b0fb4ed187c9f283d7bc3132 (127.0.98.1:37491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e145dde20c941cbb245b82cd2f21c6d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37491: connect: Connection refused (error 111)
I20260926 07:25:30.558745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 7629
I20260926 07:25:30.564332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 6986
I20260926 07:25:30.570014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 7089
2026-09-26T07:25:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7798 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:25:30 dist-test-slave-g04g krb5kdc[7895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:25:30 dist-test-slave-g04g krb5kdc[7895](info): set up 2 sockets
Sep 26 07:25:30 dist-test-slave-g04g krb5kdc[7895](info): commencing operation
krb5kdc: starting...
W20260926 07:25:31.607774 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:25:31 dist-test-slave-g04g krb5kdc[7895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:25:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 07:25:31.734972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40599
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:31.842545 7908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:31.842813 7908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:31.842862 7908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:31.846753 7908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:31.846804 7908 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:25:31.846822 7908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:31.846844 7908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:31.846861 7908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:31.846895 7908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:31.852027 7908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40599
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:31.852970 7908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:31.853888 7908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:31.862401 7914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:31.862370 7913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:31.862534 7908 server_base.cc:1034] running on GCE node
W20260926 07:25:31.862473 7916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:31.863236 7908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:31.864372 7908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:31.865535 7908 hybrid_clock.cc:648] HybridClock initialized: now 1790407531865521 us; error 26 us; skew 500 ppm
Sep 26 07:25:31 dist-test-slave-g04g krb5kdc[7895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407531, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:31.868295 7908 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 07:25:31.869136 7908 webserver.cc:460] Webserver started at http://127.0.98.62:43841/ using document root <none> and password file <none>
I20260926 07:25:31.869659 7908 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:31.869717 7908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:31.869885 7908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:31.871958 7908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "b701b88e580c465199a1b51d30a77c7d"
format_stamp: "Formatted at 2026-09-26 07:25:31 on dist-test-slave-g04g"
server_key: "d14676ab8b1296a31a8767e3fc3ccfcf"
server_key_iv: "3bad3c2e534be75373822b31a41d2abc"
server_key_version: "encryptionkey@0"
I20260926 07:25:31.872447 7908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "b701b88e580c465199a1b51d30a77c7d"
format_stamp: "Formatted at 2026-09-26 07:25:31 on dist-test-slave-g04g"
server_key: "d14676ab8b1296a31a8767e3fc3ccfcf"
server_key_iv: "3bad3c2e534be75373822b31a41d2abc"
server_key_version: "encryptionkey@0"
I20260926 07:25:31.875943 7908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:31.878262 7923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:31.879529 7908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:31.879658 7908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "b701b88e580c465199a1b51d30a77c7d"
format_stamp: "Formatted at 2026-09-26 07:25:31 on dist-test-slave-g04g"
server_key: "d14676ab8b1296a31a8767e3fc3ccfcf"
server_key_iv: "3bad3c2e534be75373822b31a41d2abc"
server_key_version: "encryptionkey@0"
I20260926 07:25:31.879760 7908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:31.907984 7908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:31.911177 7908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:31.911394 7908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:31.919269 7908 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40599
I20260926 07:25:31.919274 7985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40599 every 8 connection(s)
I20260926 07:25:31.920217 7908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:31.920977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 7908
I20260926 07:25:31.921125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:31.921398 392 external_mini_cluster.cc:1428] Setting key fb6c5c81a138bc8930ad4dc9d616e5e5
I20260926 07:25:31.923345 7986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:31.928642 7986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d: Bootstrap starting.
I20260926 07:25:31.930795 7986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d: Neither blocks nor log segments found. Creating new log.
Sep 26 07:25:31 dist-test-slave-g04g krb5kdc[7895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:31.931648 7986 log.cc:824] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:31.934111 7986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d: No bootstrap required, opened a new log
I20260926 07:25:31.937629 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:31.922727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.0.98.62:40599)
0926 07:25:31.923347 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:31.923363 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:31.923402 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:25:31.924160 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:31.924179 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:31.924523 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:31.924898 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:31.925785 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:31.926855 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:31.928334 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:31.928882 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:31.929049 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:31.931570 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:31.931576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:31.931592 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:31.931621 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:31.934355 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:31.934808 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:31.934813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:31.934819 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:31.934964 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:31.935368 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:31.935371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:31.935373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:31.935866 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:31.936049 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:31.936370 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 07:25:31.936662 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":126,"threads_started":1}
I20260926 07:25:31.937970 7986 raft_consensus.cc:348] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b701b88e580c465199a1b51d30a77c7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40599 } }
I20260926 07:25:31.938232 7986 raft_consensus.cc:374] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:31.938287 7986 raft_consensus.cc:729] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b701b88e580c465199a1b51d30a77c7d, State: Initialized, Role: FOLLOWER
I20260926 07:25:31.938776 7986 consensus_queue.cc:260] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b701b88e580c465199a1b51d30a77c7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40599 } }
I20260926 07:25:31.938905 7986 raft_consensus.cc:388] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:31.938988 7986 raft_consensus.cc:482] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:31.939126 7986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:31.940274 7986 raft_consensus.cc:504] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b701b88e580c465199a1b51d30a77c7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40599 } }
I20260926 07:25:31.940557 7986 leader_election.cc:302] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b701b88e580c465199a1b51d30a77c7d; no voters:
I20260926 07:25:31.940851 7986 leader_election.cc:288] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:31.940917 7991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:31.941112 7991 raft_consensus.cc:686] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [term 1 LEADER]: Becoming Leader. State: Replica: b701b88e580c465199a1b51d30a77c7d, State: Running, Role: LEADER
I20260926 07:25:31.941408 7991 consensus_queue.cc:237] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b701b88e580c465199a1b51d30a77c7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40599 } }
I20260926 07:25:31.942118 7986 sys_catalog.cc:564] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:31.943212 7993 sys_catalog.cc:455] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b701b88e580c465199a1b51d30a77c7d. Latest consensus state: current_term: 1 leader_uuid: "b701b88e580c465199a1b51d30a77c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b701b88e580c465199a1b51d30a77c7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40599 } } }
I20260926 07:25:31.943354 7993 sys_catalog.cc:458] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:31.943691 7992 sys_catalog.cc:455] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b701b88e580c465199a1b51d30a77c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b701b88e580c465199a1b51d30a77c7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40599 } } }
I20260926 07:25:31.943755 8000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:31.943845 7992 sys_catalog.cc:458] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:31.947077 8000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:31.952313 8000 catalog_manager.cc:1347] Generated new cluster ID: 4dc3205997ef4703a0c26dda4d70373d
I20260926 07:25:31.952391 8000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:31.980633 8000 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:31.981586 8000 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:31.990250 8000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d: Generated new TSK 0
I20260926 07:25:31.990981 8000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 07:25:32.030301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40599
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:32.153373 8012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:32.153645 8012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:32.153735 8012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:32.157743 8012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:32.157824 8012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:32.157894 8012 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:25:32.157974 8012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:32.162949 8012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:32.163942 8012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:32.164903 8012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:32.173952 8020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:32.174047 8017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:32.173952 8018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:32.174278 8012 server_base.cc:1034] running on GCE node
I20260926 07:25:32.174693 8012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:32.175393 8012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:32.176571 8012 hybrid_clock.cc:648] HybridClock initialized: now 1790407532176536 us; error 47 us; skew 500 ppm
Sep 26 07:25:32 dist-test-slave-g04g krb5kdc[7895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407532, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:32.179168 8012 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 07:25:32.179945 8012 webserver.cc:460] Webserver started at http://127.0.98.1:39543/ using document root <none> and password file <none>
I20260926 07:25:32.180469 8012 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:32.180553 8012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:32.180769 8012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:32.182724 8012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "91b18845f2ad4985a6f64b755f73700c"
format_stamp: "Formatted at 2026-09-26 07:25:32 on dist-test-slave-g04g"
server_key: "9746c1a0812bc5a76a1f648643190aa4"
server_key_iv: "954ccd2ad1f9f86befc02a44a274d8f0"
server_key_version: "encryptionkey@0"
I20260926 07:25:32.183276 8012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "91b18845f2ad4985a6f64b755f73700c"
format_stamp: "Formatted at 2026-09-26 07:25:32 on dist-test-slave-g04g"
server_key: "9746c1a0812bc5a76a1f648643190aa4"
server_key_iv: "954ccd2ad1f9f86befc02a44a274d8f0"
server_key_version: "encryptionkey@0"
I20260926 07:25:32.186761 8012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:25:32.189100 8027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:32.190253 8012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:25:32.190392 8012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "91b18845f2ad4985a6f64b755f73700c"
format_stamp: "Formatted at 2026-09-26 07:25:32 on dist-test-slave-g04g"
server_key: "9746c1a0812bc5a76a1f648643190aa4"
server_key_iv: "954ccd2ad1f9f86befc02a44a274d8f0"
server_key_version: "encryptionkey@0"
I20260926 07:25:32.190497 8012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:32.209331 8012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:32.212235 8012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:32.212432 8012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:32.213829 8012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:32.213898 8012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:32.213968 8012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:32.214031 8012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:25:32 dist-test-slave-g04g krb5kdc[7895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407532, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:32.224920 8012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45909
I20260926 07:25:32.225009 8142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45909 every 8 connection(s)
I20260926 07:25:32.225986 8012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:32.226800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 8012
I20260926 07:25:32.226929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:25:32.227244 392 external_mini_cluster.cc:1428] Setting key bd6ceb8aab01ef8d40354eac6933208e
I20260926 07:25:32.228359 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.214551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41131 (local address 127.0.98.62:40599)
0926 07:25:32.214663 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.214668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.215672 (+ 1004us) server_negotiation.cc:407] Connection header received
0926 07:25:32.216669 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.216673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.216728 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.216810 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:32.218428 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.219282 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.220565 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.220883 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.220997 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.223742 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:32.223746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:32.223748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:32.223770 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:32.225494 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.226411 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.226416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.226417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.226539 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.227022 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.227026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.227028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.227303 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:32.227412 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.228012 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.228168 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 07:25:32.229071 8036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.215070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40599 (local address 127.0.98.1:41131)
0926 07:25:32.215530 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.215570 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.216475 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.217044 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.217054 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.217560 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:32.218228 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.218251 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.219406 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.219410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.220428 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.220438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.221041 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.221045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.221222 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.221815 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:32.221834 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:32.223549 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:32.225762 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.225768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.225780 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.226256 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.226678 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.226681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.226683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.226886 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.227417 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:32.227425 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:32.227722 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.227961 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.228287 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":137,"threads_started":1}
I20260926 07:25:32.230614 8143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40599
I20260926 07:25:32.230757 8143 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:32.231352 8143 heartbeater.cc:511] Master 127.0.98.62:40599 requested a full tablet report, sending...
I20260926 07:25:32.233037 7939 ts_manager.cc:194] Registered new tserver with Master: 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909)
I20260926 07:25:32.234726 7939 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41131
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260926 07:25:32.242895 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.232792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40599 (local address 127.0.98.1:37195)
0926 07:25:32.233041 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.233061 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.233152 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.235716 (+ 2564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.235719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.235938 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:32.236081 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.236088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.237084 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.237091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.237718 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.237726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.238237 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.238239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.238304 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.238865 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:32.238876 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:32.239312 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:32.241363 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.241368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.241372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.241644 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.241911 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.241915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.241919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.242017 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.242357 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:32.242360 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:32.242464 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.242564 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.242702 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260926 07:25:32.242913 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.235145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37195 (local address 127.0.98.62:40599)
0926 07:25:32.235252 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.235256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.235271 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:25:32.235306 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.235309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.235361 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.235442 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:32.236211 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.236937 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.237822 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.238138 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.238218 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.239459 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:32.239461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:32.239464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:32.239482 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:32.241236 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.241748 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.241751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.241752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.241824 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.242112 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.242114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.242115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.242264 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:32.242347 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.242617 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.242808 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 07:25:32.258621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40599
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:32.366719 8152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:32.366978 8152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:32.367022 8152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:32.371094 8152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:32.371197 8152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:32.371277 8152 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:25:32.371361 8152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:32.376458 8152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:32.377457 8152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:32.378456 8152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:32.387514 8157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:32.387518 8160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:32.387550 8158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:32.387579 8152 server_base.cc:1034] running on GCE node
I20260926 07:25:32.388103 8152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:32.388761 8152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:32.389943 8152 hybrid_clock.cc:648] HybridClock initialized: now 1790407532389934 us; error 29 us; skew 500 ppm
Sep 26 07:25:32 dist-test-slave-g04g krb5kdc[7895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407532, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:32.392566 8152 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 07:25:32.393448 8152 webserver.cc:460] Webserver started at http://127.0.98.2:40141/ using document root <none> and password file <none>
I20260926 07:25:32.394003 8152 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:32.394081 8152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:32.394356 8152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:32.396440 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "0d3989f13a8f4dc69973ebef84c00dd4"
format_stamp: "Formatted at 2026-09-26 07:25:32 on dist-test-slave-g04g"
server_key: "f7dff62bdbe70e0a04bc2d5c0b579147"
server_key_iv: "f0038d9c898f48b2dcc544c63889432b"
server_key_version: "encryptionkey@0"
I20260926 07:25:32.397014 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0d3989f13a8f4dc69973ebef84c00dd4"
format_stamp: "Formatted at 2026-09-26 07:25:32 on dist-test-slave-g04g"
server_key: "f7dff62bdbe70e0a04bc2d5c0b579147"
server_key_iv: "f0038d9c898f48b2dcc544c63889432b"
server_key_version: "encryptionkey@0"
I20260926 07:25:32.400530 8152 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:25:32.402961 8167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:32.404165 8152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:32.404296 8152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "0d3989f13a8f4dc69973ebef84c00dd4"
format_stamp: "Formatted at 2026-09-26 07:25:32 on dist-test-slave-g04g"
server_key: "f7dff62bdbe70e0a04bc2d5c0b579147"
server_key_iv: "f0038d9c898f48b2dcc544c63889432b"
server_key_version: "encryptionkey@0"
I20260926 07:25:32.404403 8152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:32.424615 8152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:32.427623 8152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:32.427843 8152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:32.429337 8152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:32.429409 8152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:32.429481 8152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:32.429515 8152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:25:32 dist-test-slave-g04g krb5kdc[7895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407532, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:32.440949 8152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37847
I20260926 07:25:32.441138 8282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37847 every 8 connection(s)
I20260926 07:25:32.442018 8152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:25:32.443897 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.430026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60451 (local address 127.0.98.62:40599)
0926 07:25:32.430164 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.430167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.431272 (+ 1105us) server_negotiation.cc:407] Connection header received
0926 07:25:32.432379 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.432382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.432441 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.432503 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:32.433847 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.434616 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.435945 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.436251 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.436359 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.439802 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:32.439808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:32.439813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:32.439841 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:32.441659 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.442175 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.442178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.442180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.442266 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.442659 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.442662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.442664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.442904 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:32.443008 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.443671 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.443772 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:25:32.444607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 8152
I20260926 07:25:32.444721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:25:32.444869 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.430570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40599 (local address 127.0.98.2:60451)
0926 07:25:32.431136 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.431177 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.432125 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.432942 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.432948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.433256 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:32.433700 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.433711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.434736 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.434740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.435822 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.435831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.436405 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.436409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.436604 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.437185 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:32.437203 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:32.439591 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:32.441764 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.441769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.441779 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.442022 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.442381 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.442385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.442387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.442545 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.443099 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:32.443106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:32.443402 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.443618 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.443886 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":134,"threads_started":1}
I20260926 07:25:32.445057 392 external_mini_cluster.cc:1428] Setting key ddf5dc01f1cd24202e960776217dbb6d
I20260926 07:25:32.446157 8283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40599
I20260926 07:25:32.446326 8283 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:32.446864 8283 heartbeater.cc:511] Master 127.0.98.62:40599 requested a full tablet report, sending...
I20260926 07:25:32.447919 7939 ts_manager.cc:194] Registered new tserver with Master: 0d3989f13a8f4dc69973ebef84c00dd4 (127.0.98.2:37847)
I20260926 07:25:32.448704 7939 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60451
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 07:25:32.457818 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.448511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58551 (local address 127.0.98.62:40599)
0926 07:25:32.448604 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.448608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.449143 (+ 535us) server_negotiation.cc:407] Connection header received
0926 07:25:32.449249 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.449253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.449305 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.449359 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:32.451532 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.452381 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.453184 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.453550 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.453669 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.454710 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:32.454714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:32.454717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:32.454744 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:32.456272 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.456698 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.456700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.456702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.456801 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.457117 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.457119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.457120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.457301 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:32.457376 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.457621 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.457702 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:25:32.457962 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.448679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40599 (local address 127.0.98.2:58551)
0926 07:25:32.449049 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.449062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.449154 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.451186 (+ 2032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.451188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.451347 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:32.451435 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.451440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.452522 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.452525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.453072 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.453079 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.453673 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.453676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.453744 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.454262 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:32.454273 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:32.454572 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:32.456387 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.456390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.456391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.456610 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.456919 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.456925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.456930 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.457032 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.457404 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:32.457406 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:32.457481 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.457570 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.457681 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":91,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 07:25:32.474340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40599
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:32.582623 8292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:32.582916 8292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:32.583002 8292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:32.587352 8292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:32.587409 8292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:32.587448 8292 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:25:32.587481 8292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:32.592657 8292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:32.593693 8292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:32.594713 8292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:32.604103 8297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:32.604077 8300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:32.604084 8298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:32.604497 8292 server_base.cc:1034] running on GCE node
I20260926 07:25:32.604826 8292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:32.605342 8292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:32.606496 8292 hybrid_clock.cc:648] HybridClock initialized: now 1790407532606462 us; error 46 us; skew 500 ppm
Sep 26 07:25:32 dist-test-slave-g04g krb5kdc[7895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407532, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:32.609411 8292 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 07:25:32.610280 8292 webserver.cc:460] Webserver started at http://127.0.98.3:45989/ using document root <none> and password file <none>
I20260926 07:25:32.610814 8292 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:32.610873 8292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:32.611163 8292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:32.613176 8292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "ac6edff32e5f48059f45b45dff9487ed"
format_stamp: "Formatted at 2026-09-26 07:25:32 on dist-test-slave-g04g"
server_key: "08060351e601b24294f2e8102db8c221"
server_key_iv: "09d336f8e0ac2c8ff16472d6e8585bfb"
server_key_version: "encryptionkey@0"
I20260926 07:25:32.613719 8292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ac6edff32e5f48059f45b45dff9487ed"
format_stamp: "Formatted at 2026-09-26 07:25:32 on dist-test-slave-g04g"
server_key: "08060351e601b24294f2e8102db8c221"
server_key_iv: "09d336f8e0ac2c8ff16472d6e8585bfb"
server_key_version: "encryptionkey@0"
I20260926 07:25:32.617311 8292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:25:32.619634 8307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:32.621398 8292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:32.621531 8292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "ac6edff32e5f48059f45b45dff9487ed"
format_stamp: "Formatted at 2026-09-26 07:25:32 on dist-test-slave-g04g"
server_key: "08060351e601b24294f2e8102db8c221"
server_key_iv: "09d336f8e0ac2c8ff16472d6e8585bfb"
server_key_version: "encryptionkey@0"
I20260926 07:25:32.621660 8292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:32.639650 8292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:32.643169 8292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:32.643357 8292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:32.644819 8292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:32.644888 8292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:32.644958 8292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:32.645007 8292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:25:32 dist-test-slave-g04g krb5kdc[7895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407532, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:32.655795 8292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44619
I20260926 07:25:32.655869 8422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44619 every 8 connection(s)
I20260926 07:25:32.656872 8292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:32.657883 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.645507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52491 (local address 127.0.98.62:40599)
0926 07:25:32.645647 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.645652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.646466 (+ 814us) server_negotiation.cc:407] Connection header received
0926 07:25:32.647535 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.647538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.647602 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.647747 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:32.648988 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.649871 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.651106 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.651329 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.651439 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.653979 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:32.653986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:32.653993 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:32.654028 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:32.655819 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.656362 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.656365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.656366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.656442 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.656802 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.656804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.656806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.657019 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:32.657095 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.657672 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.657770 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:25:32.658695 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.645898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40599 (local address 127.0.98.3:52491)
0926 07:25:32.646325 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.646373 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.647315 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.648040 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.648051 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.648401 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:32.648850 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.648863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.649998 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.650002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.650943 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.650952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.651483 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.651487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.651656 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.652289 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:32.652305 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:32.653807 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:32.655947 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.655952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.655965 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.656266 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.656540 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.656545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.656550 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.656714 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.657100 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:32.657105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:32.657408 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.657618 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.657919 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":123,"threads_started":1}
I20260926 07:25:32.659888 8423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40599
I20260926 07:25:32.660039 8423 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:32.660468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 8292
I20260926 07:25:32.660564 8423 heartbeater.cc:511] Master 127.0.98.62:40599 requested a full tablet report, sending...
I20260926 07:25:32.660604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:32.660943 392 external_mini_cluster.cc:1428] Setting key 222c297bcc2b9868bed8c23a0792e80b
I20260926 07:25:32.661727 7939 ts_manager.cc:194] Registered new tserver with Master: ac6edff32e5f48059f45b45dff9487ed (127.0.98.3:44619)
I20260926 07:25:32.662457 7939 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52491
I20260926 07:25:32.663162 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:25:32.670985 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.662417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40599 (local address 127.0.98.3:51977)
0926 07:25:32.663030 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.663044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.663193 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.663551 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.663553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.663745 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:32.664077 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.664085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.665135 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.665139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.666035 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.666044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.666750 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.666753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.666834 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.667364 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:32.667375 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:32.667671 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:32.669476 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.669479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.669481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.669663 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.669997 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.670000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.670001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.670106 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.670543 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:32.670545 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:32.670638 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.670730 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.670839 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":90,"threads_started":1}
I20260926 07:25:32.671304 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.662251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51977 (local address 127.0.98.62:40599)
0926 07:25:32.662363 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.662368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.663132 (+ 764us) server_negotiation.cc:407] Connection header received
0926 07:25:32.663319 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.663323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.663375 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.663431 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:32.664206 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.664971 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.666192 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.666627 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.666773 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.667811 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:32.667818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:32.667821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:32.667855 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:32.669347 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.669759 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.669773 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.669777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.669870 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.670215 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.670220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.670223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.670444 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:32.670552 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.670817 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.670936 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 07:25:32 dist-test-slave-g04g krb5kdc[7895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407532, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:25:32 dist-test-slave-g04g krb5kdc[7895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:32 dist-test-slave-g04g krb5kdc[7895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:25:32.688179 7989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:32.681963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55430 (local address 127.0.98.62:40599)
0926 07:25:32.682117 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.682121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.682135 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:32.682175 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.682178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.682214 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.682304 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:32.683023 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.683608 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.684560 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.684760 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.684832 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.688006 (+ 3174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55430: BlockingRecv error: recv got EOF from 127.0.0.1:55430 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 07:25:32 dist-test-slave-g04g krb5kdc[7895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407532, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:32.697734 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.689604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.0.98.62:40599)
0926 07:25:32.689710 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.689714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.689835 (+ 121us) server_negotiation.cc:407] Connection header received
0926 07:25:32.689935 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.689937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.689972 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.690037 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:32.690573 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.691453 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.692259 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.692461 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.692539 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.695159 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:32.695162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:32.695164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:32.695185 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:32.696376 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.696773 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.696776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.696777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.696842 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.697074 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.697076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.697077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.697225 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:32.697333 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.697533 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.697621 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:25:32.700712 7940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:25:32.703728 7940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:32.718259 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.712547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.0.98.1:45909)
0926 07:25:32.713370 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.713384 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.713406 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:25:32.713478 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.713484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.713669 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.713835 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:32.714252 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.715279 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.716597 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.717413 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.717522 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.717683 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.717985 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.718077 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":80,"threads_started":1}
I20260926 07:25:32.718293 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.711975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45909 (local address 127.0.0.1:46300)
0926 07:25:32.712642 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.712697 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.712829 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.713991 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.713993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.714033 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:32.714127 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.714133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.715384 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.715389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.716480 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.716488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.717542 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.717546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.717747 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.717824 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.717936 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.718099 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":134,"spinlock_wait_cycles":11904,"thread_start_us":69,"threads_started":1}
I20260926 07:25:32.718293 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.711890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44619 (local address 127.0.0.1:43986)
0926 07:25:32.712640 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.712697 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.712830 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.713602 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.713608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.713632 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:32.713751 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.713756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.715131 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.715134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.716545 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.716555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.717542 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.717546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.717726 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.717823 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.717936 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.718099 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":140,"spinlock_wait_cycles":768,"thread_start_us":66,"threads_started":1}
I20260926 07:25:32.719095 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.712238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37847 (local address 127.0.0.1:38044)
0926 07:25:32.713126 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.713138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.713228 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.713825 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.713829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.713848 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:32.713944 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.713951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.714958 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.714962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.716546 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.716555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.718678 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.718682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.718778 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.718810 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.718861 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.718932 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":153,"thread_start_us":98,"threads_started":1}
I20260926 07:25:32.718760 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.712192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.0.98.3:44619)
0926 07:25:32.712795 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.712809 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.712833 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:25:32.713025 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.713032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.713272 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.713424 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:32.713896 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.715022 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.716653 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.717410 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.717522 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.717682 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.718505 (+ 823us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.718617 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":130,"threads_started":1}
I20260926 07:25:32.719970 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.712892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38044 (local address 127.0.98.2:37847)
0926 07:25:32.713463 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.713473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.713490 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:25:32.713551 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.713555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.713694 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.713835 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:32.714059 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.714840 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.717749 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.718546 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.719644 (+ 1098us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.719769 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.719807 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.719871 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":112,"threads_started":1}
I20260926 07:25:32.721952 8356 tablet_service.cc:1467] Processing CreateTablet for tablet 2150ad1bf02845c1869a4763b45ce47e (DEFAULT_TABLE table=test-table [id=7a769d333b954f069745a0d2314c6b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:32.721956 8077 tablet_service.cc:1467] Processing CreateTablet for tablet 2150ad1bf02845c1869a4763b45ce47e (DEFAULT_TABLE table=test-table [id=7a769d333b954f069745a0d2314c6b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:32.722258 8217 tablet_service.cc:1467] Processing CreateTablet for tablet 2150ad1bf02845c1869a4763b45ce47e (DEFAULT_TABLE table=test-table [id=7a769d333b954f069745a0d2314c6b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:32.723099 8077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2150ad1bf02845c1869a4763b45ce47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:32.723102 8356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2150ad1bf02845c1869a4763b45ce47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:32.723239 8217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2150ad1bf02845c1869a4763b45ce47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:32.729853 8451 tablet_bootstrap.cc:492] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4: Bootstrap starting.
I20260926 07:25:32.731088 8452 tablet_bootstrap.cc:492] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c: Bootstrap starting.
I20260926 07:25:32.731261 8453 tablet_bootstrap.cc:492] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed: Bootstrap starting.
I20260926 07:25:32.732710 8451 tablet_bootstrap.cc:654] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:32.733151 8453 tablet_bootstrap.cc:654] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:32.733279 8452 tablet_bootstrap.cc:654] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:32.733573 8451 log.cc:824] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:32.733893 8453 log.cc:824] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:32.734052 8452 log.cc:824] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:32.735656 8451 tablet_bootstrap.cc:492] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4: No bootstrap required, opened a new log
I20260926 07:25:32.735859 8452 tablet_bootstrap.cc:492] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c: No bootstrap required, opened a new log
I20260926 07:25:32.735857 8451 ts_tablet_manager.cc:1397] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:25:32.736017 8452 ts_tablet_manager.cc:1397] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:25:32.736445 8453 tablet_bootstrap.cc:492] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed: No bootstrap required, opened a new log
I20260926 07:25:32.736632 8453 ts_tablet_manager.cc:1397] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 07:25:32.739553 8453 raft_consensus.cc:348] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } }
I20260926 07:25:32.739794 8453 raft_consensus.cc:374] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:32.739657 8451 raft_consensus.cc:348] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } }
I20260926 07:25:32.739657 8452 raft_consensus.cc:348] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } }
I20260926 07:25:32.739866 8453 raft_consensus.cc:729] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6edff32e5f48059f45b45dff9487ed, State: Initialized, Role: FOLLOWER
I20260926 07:25:32.739902 8452 raft_consensus.cc:374] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:32.739902 8451 raft_consensus.cc:374] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:32.739959 8451 raft_consensus.cc:729] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3989f13a8f4dc69973ebef84c00dd4, State: Initialized, Role: FOLLOWER
I20260926 07:25:32.740000 8452 raft_consensus.cc:729] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b18845f2ad4985a6f64b755f73700c, State: Initialized, Role: FOLLOWER
I20260926 07:25:32.740360 8453 consensus_queue.cc:260] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } }
I20260926 07:25:32.740376 8451 consensus_queue.cc:260] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } }
I20260926 07:25:32.740376 8452 consensus_queue.cc:260] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } }
I20260926 07:25:32.741020 8423 heartbeater.cc:503] Master 127.0.98.62:40599 was elected leader, sending a full tablet report...
I20260926 07:25:32.741343 8453 ts_tablet_manager.cc:1428] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:25:32.741389 8452 ts_tablet_manager.cc:1428] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 07:25:32.741690 8143 heartbeater.cc:503] Master 127.0.98.62:40599 was elected leader, sending a full tablet report...
I20260926 07:25:32.742059 8283 heartbeater.cc:503] Master 127.0.98.62:40599 was elected leader, sending a full tablet report...
I20260926 07:25:32.742345 8451 ts_tablet_manager.cc:1428] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:25:32.745677 8458 raft_consensus.cc:482] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:32.745895 8458 raft_consensus.cc:504] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } }
I20260926 07:25:32.747117 8458 leader_election.cc:288] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac6edff32e5f48059f45b45dff9487ed (127.0.98.3:44619), 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909)
I20260926 07:25:32.751165 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.747270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44619 (local address 127.0.98.2:50839)
0926 07:25:32.747520 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.747536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.747616 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.747848 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.747850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.747861 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:32.747947 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.747951 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.748935 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.748938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.749915 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.749924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.750735 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.750739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.750843 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.750867 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.750945 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.751012 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260926 07:25:32.751161 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.747490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50839 (local address 127.0.98.3:44619)
0926 07:25:32.747629 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.747634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.747648 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:32.747697 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.747703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.747756 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.747840 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:32.748040 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.748835 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.750030 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.750628 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.750701 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.750748 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.750964 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.751016 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:25:32.751695 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.747356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45909 (local address 127.0.98.2:40205)
0926 07:25:32.747665 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.747679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.747771 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.748105 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.748108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.748123 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:32.748610 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.748619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.749510 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.749513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.750447 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.750456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.751276 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.751281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.751382 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.751408 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.751503 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.751565 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":96,"threads_started":1}
I20260926 07:25:32.751699 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.747391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40205 (local address 127.0.98.1:45909)
0926 07:25:32.747513 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.747517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.747741 (+ 224us) server_negotiation.cc:407] Connection header received
0926 07:25:32.747868 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.747871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.747916 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.747960 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:32.748725 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.749404 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.750561 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.751131 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.751245 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.751310 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.751517 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.751586 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:25:32.751833 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2150ad1bf02845c1869a4763b45ce47e" candidate_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6edff32e5f48059f45b45dff9487ed" is_pre_election: true
I20260926 07:25:32.752116 8377 raft_consensus.cc:2454] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d3989f13a8f4dc69973ebef84c00dd4 in term 0.
I20260926 07:25:32.752230 8097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2150ad1bf02845c1869a4763b45ce47e" candidate_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b18845f2ad4985a6f64b755f73700c" is_pre_election: true
I20260926 07:25:32.752470 8097 raft_consensus.cc:2454] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d3989f13a8f4dc69973ebef84c00dd4 in term 0.
I20260926 07:25:32.752573 8171 leader_election.cc:302] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3989f13a8f4dc69973ebef84c00dd4, ac6edff32e5f48059f45b45dff9487ed; no voters:
I20260926 07:25:32.753026 8458 raft_consensus.cc:2781] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:32.753094 8458 raft_consensus.cc:482] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:32.753147 8458 raft_consensus.cc:3037] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:32.754114 8458 raft_consensus.cc:504] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } }
I20260926 07:25:32.754448 8458 leader_election.cc:288] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [CANDIDATE]: Term 1 election: Requested vote from peers ac6edff32e5f48059f45b45dff9487ed (127.0.98.3:44619), 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909)
I20260926 07:25:32.754763 8097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2150ad1bf02845c1869a4763b45ce47e" candidate_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b18845f2ad4985a6f64b755f73700c"
I20260926 07:25:32.754756 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2150ad1bf02845c1869a4763b45ce47e" candidate_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6edff32e5f48059f45b45dff9487ed"
I20260926 07:25:32.754876 8097 raft_consensus.cc:3037] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:32.754879 8377 raft_consensus.cc:3037] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:32.756078 8097 raft_consensus.cc:2454] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d3989f13a8f4dc69973ebef84c00dd4 in term 1.
I20260926 07:25:32.756078 8377 raft_consensus.cc:2454] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d3989f13a8f4dc69973ebef84c00dd4 in term 1.
I20260926 07:25:32.756356 8171 leader_election.cc:302] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d3989f13a8f4dc69973ebef84c00dd4, 91b18845f2ad4985a6f64b755f73700c, ac6edff32e5f48059f45b45dff9487ed; no voters:
I20260926 07:25:32.756544 8458 raft_consensus.cc:2781] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:32.756743 8458 raft_consensus.cc:686] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 0d3989f13a8f4dc69973ebef84c00dd4, State: Running, Role: LEADER
I20260926 07:25:32.757043 8458 consensus_queue.cc:237] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } }
I20260926 07:25:32.760244 7937 catalog_manager.cc:5495] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d3989f13a8f4dc69973ebef84c00dd4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } health_report { overall_health: HEALTHY } } }
I20260926 07:25:32.770174 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.766994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38058 (local address 127.0.98.2:37847)
0926 07:25:32.767135 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.767140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.767160 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:25:32.767249 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.767251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.767297 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.767383 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:32.767546 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.768027 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.768901 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.769112 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.769191 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.769438 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:32.769855 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:32.769938 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.770022 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.770067 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:25:32.775701 8097 raft_consensus.cc:1261] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Refusing update from remote peer 0d3989f13a8f4dc69973ebef84c00dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:32.776005 8377 raft_consensus.cc:1261] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Refusing update from remote peer 0d3989f13a8f4dc69973ebef84c00dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:32.776459 8461 consensus_queue.cc:1035] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:32.776654 8458 consensus_queue.cc:1035] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:32.785051 8466 mvcc.cc:205] Tried to move back new op lower bound from 7333509254242738176 to 7333509254175551488. Current Snapshot: MvccSnapshot[applied={T|T < 7333509254242738176}]
I20260926 07:25:32.785113 8465 mvcc.cc:205] Tried to move back new op lower bound from 7333509254242738176 to 7333509254175551488. Current Snapshot: MvccSnapshot[applied={T|T < 7333509254242738176}]
I20260926 07:25:32.793012 7937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:32.793207 7937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:32.795619 7937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d: Sending DeleteTablet for 3 replicas of tablet 2150ad1bf02845c1869a4763b45ce47e
I20260926 07:25:32.796336 8356 tablet_service.cc:1514] Processing DeleteTablet for tablet 2150ad1bf02845c1869a4763b45ce47e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43986
I20260926 07:25:32.796378 8077 tablet_service.cc:1514] Processing DeleteTablet for tablet 2150ad1bf02845c1869a4763b45ce47e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46300
I20260926 07:25:32.796805 8217 tablet_service.cc:1514] Processing DeleteTablet for tablet 2150ad1bf02845c1869a4763b45ce47e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38044
I20260926 07:25:32.796876 8474 tablet_replica.cc:330] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c: stopping tablet replica
I20260926 07:25:32.797168 8474 raft_consensus.cc:2229] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:32.797317 8475 tablet_replica.cc:330] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4: stopping tablet replica
I20260926 07:25:32.797487 8474 raft_consensus.cc:2258] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:32.797703 8475 raft_consensus.cc:2229] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:25:32.797827 7937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:25:32.798213 7937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:32.798401 8475 raft_consensus.cc:2258] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:32.798905 8474 ts_tablet_manager.cc:1905] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:32.801558 8475 ts_tablet_manager.cc:1905] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:32.802116 8474 ts_tablet_manager.cc:1918] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:32.802205 8474 log.cc:1196] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/wal/wals/2150ad1bf02845c1869a4763b45ce47e
I20260926 07:25:32.802510 8474 ts_tablet_manager.cc:1939] T 2150ad1bf02845c1869a4763b45ce47e P 91b18845f2ad4985a6f64b755f73700c: Deleting consensus metadata
I20260926 07:25:32.802991 8473 tablet_replica.cc:330] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed: stopping tablet replica
I20260926 07:25:32.802977 8077 tablet_service.cc:1467] Processing CreateTablet for tablet 20dec42974084efcb6ed9679f81931a7 (DEFAULT_TABLE table=test-table [id=ce2a5593cd0c49d08648898b77f977ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:32.803074 8217 tablet_service.cc:1467] Processing CreateTablet for tablet 20dec42974084efcb6ed9679f81931a7 (DEFAULT_TABLE table=test-table [id=ce2a5593cd0c49d08648898b77f977ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:32.803227 8077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20dec42974084efcb6ed9679f81931a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:32.803308 8473 raft_consensus.cc:2229] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:32.803349 8217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20dec42974084efcb6ed9679f81931a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:32.803589 8473 raft_consensus.cc:2258] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:32.804289 8356 tablet_service.cc:1467] Processing CreateTablet for tablet 20dec42974084efcb6ed9679f81931a7 (DEFAULT_TABLE table=test-table [id=ce2a5593cd0c49d08648898b77f977ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:32.804540 8356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20dec42974084efcb6ed9679f81931a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:32.804747 7926 catalog_manager.cc:4841] TS 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909): tablet 2150ad1bf02845c1869a4763b45ce47e (table test-table [id=7a769d333b954f069745a0d2314c6b33]) successfully deleted
I20260926 07:25:32.805420 8452 tablet_bootstrap.cc:492] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c: Bootstrap starting.
I20260926 07:25:32.806339 8452 tablet_bootstrap.cc:654] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:32.806849 8453 tablet_bootstrap.cc:492] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed: Bootstrap starting.
I20260926 07:25:32.806876 8451 tablet_bootstrap.cc:492] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4: Bootstrap starting.
I20260926 07:25:32.807845 8453 tablet_bootstrap.cc:654] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:32.808322 8473 ts_tablet_manager.cc:1905] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:32.808504 8451 tablet_bootstrap.cc:654] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:32.809456 8453 tablet_bootstrap.cc:492] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed: No bootstrap required, opened a new log
I20260926 07:25:32.809542 8453 ts_tablet_manager.cc:1397] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:25:32.810009 8453 raft_consensus.cc:348] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:32.810132 8453 raft_consensus.cc:374] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:32.810168 8453 raft_consensus.cc:729] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6edff32e5f48059f45b45dff9487ed, State: Initialized, Role: FOLLOWER
I20260926 07:25:32.810312 8453 consensus_queue.cc:260] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:32.810544 8453 ts_tablet_manager.cc:1428] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:32.810657 8473 ts_tablet_manager.cc:1918] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:32.810741 8473 log.cc:1196] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/2150ad1bf02845c1869a4763b45ce47e
I20260926 07:25:32.811003 8475 ts_tablet_manager.cc:1918] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:32.811049 8473 ts_tablet_manager.cc:1939] T 2150ad1bf02845c1869a4763b45ce47e P ac6edff32e5f48059f45b45dff9487ed: Deleting consensus metadata
I20260926 07:25:32.811105 8475 log.cc:1196] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/2150ad1bf02845c1869a4763b45ce47e
I20260926 07:25:32.812106 8475 ts_tablet_manager.cc:1939] T 2150ad1bf02845c1869a4763b45ce47e P 0d3989f13a8f4dc69973ebef84c00dd4: Deleting consensus metadata
I20260926 07:25:32.812152 7927 catalog_manager.cc:4841] TS ac6edff32e5f48059f45b45dff9487ed (127.0.98.3:44619): tablet 2150ad1bf02845c1869a4763b45ce47e (table test-table [id=7a769d333b954f069745a0d2314c6b33]) successfully deleted
I20260926 07:25:32.813123 7924 catalog_manager.cc:4841] TS 0d3989f13a8f4dc69973ebef84c00dd4 (127.0.98.2:37847): tablet 2150ad1bf02845c1869a4763b45ce47e (table test-table [id=7a769d333b954f069745a0d2314c6b33]) successfully deleted
I20260926 07:25:32.813712 8451 tablet_bootstrap.cc:492] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4: No bootstrap required, opened a new log
I20260926 07:25:32.813818 8451 ts_tablet_manager.cc:1397] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 07:25:32.814079 8452 tablet_bootstrap.cc:492] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c: No bootstrap required, opened a new log
I20260926 07:25:32.814172 8452 ts_tablet_manager.cc:1397] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260926 07:25:32.814266 8451 raft_consensus.cc:348] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:32.814390 8451 raft_consensus.cc:374] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:32.814428 8451 raft_consensus.cc:729] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3989f13a8f4dc69973ebef84c00dd4, State: Initialized, Role: FOLLOWER
I20260926 07:25:32.814559 8451 consensus_queue.cc:260] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:32.814829 8451 ts_tablet_manager.cc:1428] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:32.814920 8452 raft_consensus.cc:348] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:32.815065 8452 raft_consensus.cc:374] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:32.815109 8452 raft_consensus.cc:729] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b18845f2ad4985a6f64b755f73700c, State: Initialized, Role: FOLLOWER
I20260926 07:25:32.815227 8452 consensus_queue.cc:260] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:32.815479 8452 ts_tablet_manager.cc:1428] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
W20260926 07:25:32.907868 8424 tablet.cc:2369] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:25:32.931602 8457 raft_consensus.cc:482] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:32.931762 8457 raft_consensus.cc:504] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:32.932914 8457 leader_election.cc:288] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909), 0d3989f13a8f4dc69973ebef84c00dd4 (127.0.98.2:37847)
I20260926 07:25:32.937770 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.933528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37919 (local address 127.0.98.2:37847)
0926 07:25:32.933648 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.933653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.934007 (+ 354us) server_negotiation.cc:407] Connection header received
0926 07:25:32.934192 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.934195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.934242 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.934293 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:32.934688 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.935422 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.936817 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.937219 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.937339 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.937406 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.937580 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.937646 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:25:32.937788 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.933206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37847 (local address 127.0.98.3:37919)
0926 07:25:32.933906 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.933922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.934073 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.934420 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.934422 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.934437 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:32.934548 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.934553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.935547 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.935550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.936706 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.936713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.937341 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.937345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.937444 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.937473 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.937561 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.937641 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":83,"threads_started":1}
I20260926 07:25:32.938011 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.933077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45909 (local address 127.0.98.3:57789)
0926 07:25:32.933346 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.933362 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.933470 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.934063 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.934066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.934078 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:32.934216 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.934223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.935273 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.935276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.936550 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.936560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.937585 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.937588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.937724 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.937749 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.937825 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.937903 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260926 07:25:32.938283 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.933637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57789 (local address 127.0.98.1:45909)
0926 07:25:32.933744 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.933750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.933765 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:25:32.933818 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.933821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.933866 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.933943 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:32.934342 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.935147 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.936712 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.937328 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.937444 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.937517 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.938095 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.938171 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 07:25:32.938697 8097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dec42974084efcb6ed9679f81931a7" candidate_uuid: "ac6edff32e5f48059f45b45dff9487ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b18845f2ad4985a6f64b755f73700c" is_pre_election: true
I20260926 07:25:32.938755 8237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dec42974084efcb6ed9679f81931a7" candidate_uuid: "ac6edff32e5f48059f45b45dff9487ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" is_pre_election: true
I20260926 07:25:32.938861 8097 raft_consensus.cc:2454] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6edff32e5f48059f45b45dff9487ed in term 0.
I20260926 07:25:32.938967 8237 raft_consensus.cc:2454] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6edff32e5f48059f45b45dff9487ed in term 0.
I20260926 07:25:32.939287 8310 leader_election.cc:302] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91b18845f2ad4985a6f64b755f73700c, ac6edff32e5f48059f45b45dff9487ed; no voters:
I20260926 07:25:32.939543 8457 raft_consensus.cc:2781] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:32.939621 8457 raft_consensus.cc:482] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:32.939668 8457 raft_consensus.cc:3037] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:32.940626 8457 raft_consensus.cc:504] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:32.940994 8457 leader_election.cc:288] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [CANDIDATE]: Term 1 election: Requested vote from peers 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909), 0d3989f13a8f4dc69973ebef84c00dd4 (127.0.98.2:37847)
I20260926 07:25:32.941298 8097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dec42974084efcb6ed9679f81931a7" candidate_uuid: "ac6edff32e5f48059f45b45dff9487ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b18845f2ad4985a6f64b755f73700c"
I20260926 07:25:32.941305 8237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20dec42974084efcb6ed9679f81931a7" candidate_uuid: "ac6edff32e5f48059f45b45dff9487ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3989f13a8f4dc69973ebef84c00dd4"
I20260926 07:25:32.941413 8097 raft_consensus.cc:3037] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:32.941551 8237 raft_consensus.cc:3037] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:32.942498 8237 raft_consensus.cc:2454] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6edff32e5f48059f45b45dff9487ed in term 1.
I20260926 07:25:32.942718 8097 raft_consensus.cc:2454] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6edff32e5f48059f45b45dff9487ed in term 1.
I20260926 07:25:32.942791 8308 leader_election.cc:302] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3989f13a8f4dc69973ebef84c00dd4, ac6edff32e5f48059f45b45dff9487ed; no voters:
W20260926 07:25:32.943120 8284 tablet.cc:2369] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:25:32.943608 8457 raft_consensus.cc:2781] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:32.943848 8457 raft_consensus.cc:686] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [term 1 LEADER]: Becoming Leader. State: Replica: ac6edff32e5f48059f45b45dff9487ed, State: Running, Role: LEADER
I20260926 07:25:32.944104 8457 consensus_queue.cc:237] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:32.946424 7939 catalog_manager.cc:5495] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6edff32e5f48059f45b45dff9487ed (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ac6edff32e5f48059f45b45dff9487ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } health_report { overall_health: HEALTHY } } }
W20260926 07:25:32.978259 8144 tablet.cc:2369] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 07:25:32 dist-test-slave-g04g krb5kdc[7895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407531, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:32.996707 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.988732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40599 (local address 127.0.0.1:55444)
0926 07:25:32.989009 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:32.989022 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:32.989133 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:32.989361 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:32.989362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:32.989756 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:32.989846 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.989851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.990542 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.990545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.991219 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:32.991228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.991618 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.991621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:32.991680 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.992196 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:32.992207 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:32.993652 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:32.995379 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.995386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.995391 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.995632 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.995857 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:32.995859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:32.995861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:32.995935 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:32.996243 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:32.996248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:32.996343 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 07:25:32.996433 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 07:25:32.996546 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":111,"threads_started":1}
I20260926 07:25:32.996718 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:32.988762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.0.98.62:40599)
0926 07:25:32.988912 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:32.988917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:32.989098 (+ 181us) server_negotiation.cc:407] Connection header received
0926 07:25:32.989212 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:32.989214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:32.989256 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:32.989326 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:32.989944 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.990446 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.991316 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:32.991516 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:32.991591 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:32.993785 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:32.993788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:32.993790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:32.993809 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:32.995282 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.995713 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.995716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.995717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.995783 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:32.996005 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:32.996007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:32.996008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:32.996157 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:32.996227 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:32.996486 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 07:25:32.996596 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":68}
I20260926 07:25:33.000214 7939 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55444:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:25:33.001017 7939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:33.005713 8077 tablet_service.cc:1467] Processing CreateTablet for tablet b5836b4af01e4b708c8984285ed0bf82 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2bd9d7f64ac5483c9f0bc0ec53ac093a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:25:33.005741 8217 tablet_service.cc:1467] Processing CreateTablet for tablet b5836b4af01e4b708c8984285ed0bf82 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2bd9d7f64ac5483c9f0bc0ec53ac093a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:25:33.005851 8356 tablet_service.cc:1467] Processing CreateTablet for tablet b5836b4af01e4b708c8984285ed0bf82 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2bd9d7f64ac5483c9f0bc0ec53ac093a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:25:33.006086 8077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5836b4af01e4b708c8984285ed0bf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:33.006121 8217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5836b4af01e4b708c8984285ed0bf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:33.006158 8356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5836b4af01e4b708c8984285ed0bf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:33.008800 8453 tablet_bootstrap.cc:492] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed: Bootstrap starting.
I20260926 07:25:33.008800 8452 tablet_bootstrap.cc:492] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c: Bootstrap starting.
I20260926 07:25:33.008805 8451 tablet_bootstrap.cc:492] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4: Bootstrap starting.
I20260926 07:25:33.009721 8452 tablet_bootstrap.cc:654] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:33.009721 8453 tablet_bootstrap.cc:654] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:33.009773 8451 tablet_bootstrap.cc:654] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:33.010979 8451 tablet_bootstrap.cc:492] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4: No bootstrap required, opened a new log
I20260926 07:25:33.011123 8451 ts_tablet_manager.cc:1397] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:33.011226 8452 tablet_bootstrap.cc:492] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c: No bootstrap required, opened a new log
I20260926 07:25:33.011310 8452 ts_tablet_manager.cc:1397] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 07:25:33.011577 8451 raft_consensus.cc:348] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.011720 8451 raft_consensus.cc:374] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:33.011767 8451 raft_consensus.cc:729] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3989f13a8f4dc69973ebef84c00dd4, State: Initialized, Role: FOLLOWER
I20260926 07:25:33.011777 8452 raft_consensus.cc:348] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.011894 8452 raft_consensus.cc:374] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:33.011941 8452 raft_consensus.cc:729] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b18845f2ad4985a6f64b755f73700c, State: Initialized, Role: FOLLOWER
I20260926 07:25:33.011893 8451 consensus_queue.cc:260] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.012068 8452 consensus_queue.cc:260] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.012151 8458 tablet_replica.cc:439] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } }
I20260926 07:25:33.012212 8451 ts_tablet_manager.cc:1428] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:33.012291 8458 tablet_replica.cc:442] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:33.012334 8452 ts_tablet_manager.cc:1428] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:33.012581 8459 tablet_replica.cc:439] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } }
I20260926 07:25:33.012712 8459 tablet_replica.cc:442] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:33.013224 8453 tablet_bootstrap.cc:492] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed: No bootstrap required, opened a new log
I20260926 07:25:33.013327 8453 ts_tablet_manager.cc:1397] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 07:25:33.013803 8453 raft_consensus.cc:348] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.013927 8453 raft_consensus.cc:374] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:33.013962 8453 raft_consensus.cc:729] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6edff32e5f48059f45b45dff9487ed, State: Initialized, Role: FOLLOWER
I20260926 07:25:33.014071 8453 consensus_queue.cc:260] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.014281 8457 tablet_replica.cc:439] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } }
I20260926 07:25:33.014384 8457 tablet_replica.cc:442] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:33.014624 8453 ts_tablet_manager.cc:1428] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:33.244997 8458 raft_consensus.cc:482] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:33.245131 8458 raft_consensus.cc:504] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.245522 8458 leader_election.cc:288] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909), ac6edff32e5f48059f45b45dff9487ed (127.0.98.3:44619)
I20260926 07:25:33.245930 8097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5836b4af01e4b708c8984285ed0bf82" candidate_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b18845f2ad4985a6f64b755f73700c" is_pre_election: true
I20260926 07:25:33.246074 8097 raft_consensus.cc:2454] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d3989f13a8f4dc69973ebef84c00dd4 in term 0.
I20260926 07:25:33.246025 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5836b4af01e4b708c8984285ed0bf82" candidate_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6edff32e5f48059f45b45dff9487ed" is_pre_election: true
I20260926 07:25:33.246220 8377 raft_consensus.cc:2454] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d3989f13a8f4dc69973ebef84c00dd4 in term 0.
I20260926 07:25:33.246383 8170 leader_election.cc:302] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3989f13a8f4dc69973ebef84c00dd4, 91b18845f2ad4985a6f64b755f73700c; no voters:
I20260926 07:25:33.246573 8458 raft_consensus.cc:2781] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:33.246634 8458 raft_consensus.cc:482] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:33.246721 8458 raft_consensus.cc:3037] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:33.247788 8458 raft_consensus.cc:504] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.248073 8458 leader_election.cc:288] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909), ac6edff32e5f48059f45b45dff9487ed (127.0.98.3:44619)
I20260926 07:25:33.248435 8097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5836b4af01e4b708c8984285ed0bf82" candidate_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b18845f2ad4985a6f64b755f73700c"
I20260926 07:25:33.248564 8097 raft_consensus.cc:3037] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:33.248684 8377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5836b4af01e4b708c8984285ed0bf82" candidate_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac6edff32e5f48059f45b45dff9487ed"
I20260926 07:25:33.248818 8377 raft_consensus.cc:3037] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:33.249495 8097 raft_consensus.cc:2454] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d3989f13a8f4dc69973ebef84c00dd4 in term 1.
I20260926 07:25:33.249768 8170 leader_election.cc:302] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3989f13a8f4dc69973ebef84c00dd4, 91b18845f2ad4985a6f64b755f73700c; no voters:
I20260926 07:25:33.249809 8377 raft_consensus.cc:2454] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d3989f13a8f4dc69973ebef84c00dd4 in term 1.
I20260926 07:25:33.249965 8458 raft_consensus.cc:2781] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:33.250068 8458 raft_consensus.cc:686] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 0d3989f13a8f4dc69973ebef84c00dd4, State: Running, Role: LEADER
I20260926 07:25:33.250236 8458 consensus_queue.cc:237] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.250870 8461 tablet_replica.cc:439] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4: TxnStatusTablet state changed. Reason: New leader 0d3989f13a8f4dc69973ebef84c00dd4. Latest consensus state: current_term: 1 leader_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } }
I20260926 07:25:33.250977 8461 tablet_replica.cc:442] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:25:33.251230 8492 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:25:33.251554 8492 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:25:33.252197 7939 catalog_manager.cc:5495] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d3989f13a8f4dc69973ebef84c00dd4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } health_report { overall_health: UNKNOWN } } }
I20260926 07:25:33.305895 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:33.302452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37847 (local address 127.0.0.1:38068)
0926 07:25:33.302577 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:33.302593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:33.302692 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:33.302980 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:33.302981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:33.302989 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:25:33.303134 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:33.303139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.304117 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.304119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:33.304725 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:33.304732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.305272 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.305274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:33.305335 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:33.305373 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:25:33.305653 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:25:33.305662 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:25:33.305711 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 07:25:33.305786 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 07:25:33.305919 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:33.302556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38068 (local address 127.0.98.2:37847)
0926 07:25:33.302678 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:33.302682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:33.302697 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:25:33.302819 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:33.302823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:33.302877 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:33.302961 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:33.303236 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.304015 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.304831 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.305169 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.305270 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:33.305490 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:33.305575 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:33.305644 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:33.305737 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 07:25:33.305786 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:25:33.308686 8377 raft_consensus.cc:1261] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Refusing update from remote peer 0d3989f13a8f4dc69973ebef84c00dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:33.308873 8097 raft_consensus.cc:1261] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Refusing update from remote peer 0d3989f13a8f4dc69973ebef84c00dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:33.309083 8458 consensus_queue.cc:1035] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:33.309597 8458 consensus_queue.cc:1035] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:33.310475 8457 tablet_replica.cc:439] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed: TxnStatusTablet state changed. Reason: New leader 0d3989f13a8f4dc69973ebef84c00dd4. Latest consensus state: current_term: 1 leader_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } }
I20260926 07:25:33.310595 8457 tablet_replica.cc:442] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:33.310863 8459 tablet_replica.cc:439] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c: TxnStatusTablet state changed. Reason: New leader 0d3989f13a8f4dc69973ebef84c00dd4. Latest consensus state: current_term: 1 leader_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } }
I20260926 07:25:33.310958 8459 tablet_replica.cc:442] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:33.313011 8461 tablet_replica.cc:439] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } }
I20260926 07:25:33.313122 8461 tablet_replica.cc:442] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:25:33.313246 8461 tablet_replica.cc:439] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } }
I20260926 07:25:33.313323 8461 tablet_replica.cc:442] T b5836b4af01e4b708c8984285ed0bf82 P 0d3989f13a8f4dc69973ebef84c00dd4: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:25:33.313905 8457 tablet_replica.cc:439] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } }
I20260926 07:25:33.314028 8457 tablet_replica.cc:442] T b5836b4af01e4b708c8984285ed0bf82 P ac6edff32e5f48059f45b45dff9487ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:33.315444 8459 tablet_replica.cc:439] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } } }
I20260926 07:25:33.315548 8459 tablet_replica.cc:442] T b5836b4af01e4b708c8984285ed0bf82 P 91b18845f2ad4985a6f64b755f73700c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:25:33.317871 8495 mvcc.cc:205] Tried to move back new op lower bound from 7333509256428597248 to 7333509256194351104. Current Snapshot: MvccSnapshot[applied={T|T < 7333509256428597248 or (T in {7333509256428597248})}]
I20260926 07:25:33.321967 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:33.318061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43994 (local address 127.0.98.3:44619)
0926 07:25:33.318259 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:33.318263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:33.318424 (+ 161us) server_negotiation.cc:407] Connection header received
0926 07:25:33.318587 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:33.318590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:33.318650 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:33.318795 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:33.319046 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.319701 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.320555 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.320808 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.320915 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:33.321168 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:33.321588 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:33.321723 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:33.321758 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 07:25:33.321807 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":57,"threads_started":1}
I20260926 07:25:33.332060 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:33.328655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37847 (local address 127.0.98.3:54685)
0926 07:25:33.328959 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:33.328971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:33.329050 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:33.329260 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:33.329262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:33.329267 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:25:33.329347 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:33.329351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.330257 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.330259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:33.330850 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:33.330857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.331470 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.331476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:33.331546 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:33.331584 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:25:33.331854 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:25:33.331857 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:25:33.331900 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 07:25:33.331955 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":94,"threads_started":1}
I20260926 07:25:33.332079 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:33.328736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54685 (local address 127.0.98.2:37847)
0926 07:25:33.328829 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:33.328832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:33.329046 (+ 214us) server_negotiation.cc:407] Connection header received
0926 07:25:33.329119 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:33.329122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:33.329170 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:33.329250 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:33.329440 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.330168 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.330954 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.331334 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.331461 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:33.331703 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:33.331776 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:33.331853 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:33.331910 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 07:25:33.331959 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 07:25:33.337013 8457 consensus_queue.cc:1035] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:33.340597 8457 consensus_queue.cc:1035] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:33.362393 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:33.358368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49511 (local address 127.0.98.3:44619)
0926 07:25:33.358493 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:33.358499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:33.358512 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:25:33.358548 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:33.358551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:33.358600 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:33.358677 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:33.358924 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.359562 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.361297 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.361571 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.361690 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:33.361960 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:33.362037 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:33.362118 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:33.362213 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 07:25:33.362259 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:25:33.362439 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:33.358031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44619 (local address 127.0.98.2:49511)
0926 07:25:33.358310 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:33.358326 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:33.358426 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:33.358701 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:33.358703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:33.358711 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:25:33.358816 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:33.358821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.359700 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.359703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:33.361174 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:33.361184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.361689 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.361692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:33.361779 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:33.361830 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:25:33.362131 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:25:33.362140 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:25:33.362204 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 07:25:33.362279 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260926 07:25:33.390213 7940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:33.390389 7940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:33.392681 7940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d: Sending DeleteTablet for 3 replicas of tablet 20dec42974084efcb6ed9679f81931a7
I20260926 07:25:33.393213 8077 tablet_service.cc:1514] Processing DeleteTablet for tablet 20dec42974084efcb6ed9679f81931a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46300
I20260926 07:25:33.393534 8356 tablet_service.cc:1514] Processing DeleteTablet for tablet 20dec42974084efcb6ed9679f81931a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43986
I20260926 07:25:33.393584 8519 tablet_replica.cc:330] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c: stopping tablet replica
I20260926 07:25:33.393719 8519 raft_consensus.cc:2229] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:33.393833 8519 raft_consensus.cc:2258] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:33.394444 8217 tablet_service.cc:1514] Processing DeleteTablet for tablet 20dec42974084efcb6ed9679f81931a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38044
I20260926 07:25:33.394544 8519 ts_tablet_manager.cc:1905] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:33.399220 8520 tablet_replica.cc:330] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed: stopping tablet replica
I20260926 07:25:33.399402 8520 raft_consensus.cc:2229] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:25:33.399680 8519 ts_tablet_manager.cc:1918] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:33.399770 8519 log.cc:1196] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/wal/wals/20dec42974084efcb6ed9679f81931a7
I20260926 07:25:33.399858 8520 raft_consensus.cc:2258] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:33.400041 8519 ts_tablet_manager.cc:1939] T 20dec42974084efcb6ed9679f81931a7 P 91b18845f2ad4985a6f64b755f73700c: Deleting consensus metadata
I20260926 07:25:33.400677 8520 ts_tablet_manager.cc:1905] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:33.400725 7926 catalog_manager.cc:4841] TS 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909): tablet 20dec42974084efcb6ed9679f81931a7 (table test-table [id=ce2a5593cd0c49d08648898b77f977ee]) successfully deleted
I20260926 07:25:33.401057 8521 tablet_replica.cc:330] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4: stopping tablet replica
I20260926 07:25:33.401392 8521 raft_consensus.cc:2229] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:33.401525 8521 raft_consensus.cc:2258] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:33.402175 8521 ts_tablet_manager.cc:1905] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:33.404879 8521 ts_tablet_manager.cc:1918] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:33.404960 8521 log.cc:1196] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/20dec42974084efcb6ed9679f81931a7
I20260926 07:25:33.405227 8521 ts_tablet_manager.cc:1939] T 20dec42974084efcb6ed9679f81931a7 P 0d3989f13a8f4dc69973ebef84c00dd4: Deleting consensus metadata
I20260926 07:25:33.405874 7924 catalog_manager.cc:4841] TS 0d3989f13a8f4dc69973ebef84c00dd4 (127.0.98.2:37847): tablet 20dec42974084efcb6ed9679f81931a7 (table test-table [id=ce2a5593cd0c49d08648898b77f977ee]) successfully deleted
I20260926 07:25:33.406764 8520 ts_tablet_manager.cc:1918] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:33.406844 8520 log.cc:1196] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/20dec42974084efcb6ed9679f81931a7
I20260926 07:25:33.407131 8520 ts_tablet_manager.cc:1939] T 20dec42974084efcb6ed9679f81931a7 P ac6edff32e5f48059f45b45dff9487ed: Deleting consensus metadata
I20260926 07:25:33.407790 7927 catalog_manager.cc:4841] TS ac6edff32e5f48059f45b45dff9487ed (127.0.98.3:44619): tablet 20dec42974084efcb6ed9679f81931a7 (table test-table [id=ce2a5593cd0c49d08648898b77f977ee]) successfully deleted
Sep 26 07:25:33 dist-test-slave-g04g krb5kdc[7895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:25:33 dist-test-slave-g04g krb5kdc[7895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:33 dist-test-slave-g04g krb5kdc[7895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:25:33.431515 7989 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:33.423935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.0.98.62:40599)
0926 07:25:33.424097 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:33.424102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:33.424383 (+ 281us) server_negotiation.cc:407] Connection header received
0926 07:25:33.424527 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:33.424530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:33.424577 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:33.424657 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:33.425549 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.426150 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.427479 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.427758 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.427840 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:33.431364 (+ 3524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55460: BlockingRecv error: recv got EOF from 127.0.0.1:55460 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 07:25:33 dist-test-slave-g04g krb5kdc[7895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 07:25:33.442147 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:33.432873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.0.98.62:40599)
0926 07:25:33.432997 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:33.433000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:33.433100 (+ 100us) server_negotiation.cc:407] Connection header received
0926 07:25:33.433212 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:33.433214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:33.433252 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:33.433309 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:33.433990 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.434480 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.435605 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.435815 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.435907 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:33.438563 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:33.438567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:33.438569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:33.438593 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:33.440209 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:33.440676 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:33.440681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:33.440683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:33.440811 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:33.441144 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:33.441148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:33.441150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:33.441341 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:33.441507 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:33.441771 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 07:25:33.441931 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:25:33.447830 7940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:25:33.448318 7940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:33.453500 8077 tablet_service.cc:1467] Processing CreateTablet for tablet b9615087c4bf4b1bb4dfea49dbfec767 (DEFAULT_TABLE table=test-table [id=48ad1fa2756942e6b379c6d76d777ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:33.453665 8217 tablet_service.cc:1467] Processing CreateTablet for tablet b9615087c4bf4b1bb4dfea49dbfec767 (DEFAULT_TABLE table=test-table [id=48ad1fa2756942e6b379c6d76d777ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:33.453833 8077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9615087c4bf4b1bb4dfea49dbfec767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:33.453922 8217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9615087c4bf4b1bb4dfea49dbfec767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:33.455884 8356 tablet_service.cc:1467] Processing CreateTablet for tablet b9615087c4bf4b1bb4dfea49dbfec767 (DEFAULT_TABLE table=test-table [id=48ad1fa2756942e6b379c6d76d777ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:33.456197 8356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9615087c4bf4b1bb4dfea49dbfec767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:33.456436 8451 tablet_bootstrap.cc:492] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4: Bootstrap starting.
I20260926 07:25:33.457296 8451 tablet_bootstrap.cc:654] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:33.458432 8451 tablet_bootstrap.cc:492] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4: No bootstrap required, opened a new log
I20260926 07:25:33.458604 8451 ts_tablet_manager.cc:1397] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:33.458825 8453 tablet_bootstrap.cc:492] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed: Bootstrap starting.
I20260926 07:25:33.459080 8451 raft_consensus.cc:348] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.459241 8451 raft_consensus.cc:374] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:33.459288 8451 raft_consensus.cc:729] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3989f13a8f4dc69973ebef84c00dd4, State: Initialized, Role: FOLLOWER
I20260926 07:25:33.459409 8451 consensus_queue.cc:260] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.459704 8451 ts_tablet_manager.cc:1428] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 07:25:33.459738 8453 tablet_bootstrap.cc:654] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:33.461575 8452 tablet_bootstrap.cc:492] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c: Bootstrap starting.
I20260926 07:25:33.462344 8452 tablet_bootstrap.cc:654] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:33.463596 8452 tablet_bootstrap.cc:492] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c: No bootstrap required, opened a new log
I20260926 07:25:33.463685 8452 ts_tablet_manager.cc:1397] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:33.464115 8452 raft_consensus.cc:348] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.464241 8452 raft_consensus.cc:374] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:33.464279 8452 raft_consensus.cc:729] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b18845f2ad4985a6f64b755f73700c, State: Initialized, Role: FOLLOWER
I20260926 07:25:33.464416 8452 consensus_queue.cc:260] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.464696 8452 ts_tablet_manager.cc:1428] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:25:33.467804 8453 tablet_bootstrap.cc:492] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed: No bootstrap required, opened a new log
I20260926 07:25:33.467921 8453 ts_tablet_manager.cc:1397] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260926 07:25:33.468297 8453 raft_consensus.cc:348] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.468430 8453 raft_consensus.cc:374] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:33.468470 8453 raft_consensus.cc:729] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6edff32e5f48059f45b45dff9487ed, State: Initialized, Role: FOLLOWER
I20260926 07:25:33.468609 8453 consensus_queue.cc:260] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.468873 8453 ts_tablet_manager.cc:1428] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:25:33.502969 8480 raft_consensus.cc:482] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:33.503157 8480 raft_consensus.cc:504] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.503628 8480 leader_election.cc:288] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909), 0d3989f13a8f4dc69973ebef84c00dd4 (127.0.98.2:37847)
I20260926 07:25:33.504139 8237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9615087c4bf4b1bb4dfea49dbfec767" candidate_uuid: "ac6edff32e5f48059f45b45dff9487ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" is_pre_election: true
I20260926 07:25:33.504142 8097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9615087c4bf4b1bb4dfea49dbfec767" candidate_uuid: "ac6edff32e5f48059f45b45dff9487ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b18845f2ad4985a6f64b755f73700c" is_pre_election: true
I20260926 07:25:33.504304 8237 raft_consensus.cc:2454] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6edff32e5f48059f45b45dff9487ed in term 0.
I20260926 07:25:33.504308 8097 raft_consensus.cc:2454] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6edff32e5f48059f45b45dff9487ed in term 0.
I20260926 07:25:33.504590 8308 leader_election.cc:302] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3989f13a8f4dc69973ebef84c00dd4, ac6edff32e5f48059f45b45dff9487ed; no voters:
I20260926 07:25:33.504770 8480 raft_consensus.cc:2781] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:33.504832 8480 raft_consensus.cc:482] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:33.504884 8480 raft_consensus.cc:3037] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:33.505852 8480 raft_consensus.cc:504] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.506197 8480 leader_election.cc:288] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [CANDIDATE]: Term 1 election: Requested vote from peers 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909), 0d3989f13a8f4dc69973ebef84c00dd4 (127.0.98.2:37847)
I20260926 07:25:33.506536 8097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9615087c4bf4b1bb4dfea49dbfec767" candidate_uuid: "ac6edff32e5f48059f45b45dff9487ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b18845f2ad4985a6f64b755f73700c"
I20260926 07:25:33.506544 8237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9615087c4bf4b1bb4dfea49dbfec767" candidate_uuid: "ac6edff32e5f48059f45b45dff9487ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3989f13a8f4dc69973ebef84c00dd4"
I20260926 07:25:33.506680 8237 raft_consensus.cc:3037] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:33.506685 8097 raft_consensus.cc:3037] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:33.507689 8097 raft_consensus.cc:2454] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6edff32e5f48059f45b45dff9487ed in term 1.
I20260926 07:25:33.507956 8237 raft_consensus.cc:2454] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6edff32e5f48059f45b45dff9487ed in term 1.
I20260926 07:25:33.508255 8310 leader_election.cc:302] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91b18845f2ad4985a6f64b755f73700c, ac6edff32e5f48059f45b45dff9487ed; no voters:
I20260926 07:25:33.508458 8480 raft_consensus.cc:2781] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:33.508561 8480 raft_consensus.cc:686] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [term 1 LEADER]: Becoming Leader. State: Replica: ac6edff32e5f48059f45b45dff9487ed, State: Running, Role: LEADER
I20260926 07:25:33.508702 8480 consensus_queue.cc:237] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.510411 7940 catalog_manager.cc:5495] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6edff32e5f48059f45b45dff9487ed (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ac6edff32e5f48059f45b45dff9487ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } health_report { overall_health: HEALTHY } } }
I20260926 07:25:33.530098 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:33.526575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.0.98.3:44619)
0926 07:25:33.526700 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:33.526705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:33.526720 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:25:33.526879 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:33.526883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:33.526940 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:33.527025 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:33.527326 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.528109 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.528967 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:33.529314 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:33.529422 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:33.529661 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:33.529742 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:33.529824 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:33.529898 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 07:25:33.529949 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:25:33.532037 8097 raft_consensus.cc:1261] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Refusing update from remote peer ac6edff32e5f48059f45b45dff9487ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:33.532086 8237 raft_consensus.cc:1261] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Refusing update from remote peer ac6edff32e5f48059f45b45dff9487ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:33.532536 8480 consensus_queue.cc:1035] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:33.532720 8513 consensus_queue.cc:1035] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:33.541785 7940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:33.541920 7940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:33.544222 7940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d: Sending DeleteTablet for 3 replicas of tablet b9615087c4bf4b1bb4dfea49dbfec767
I20260926 07:25:33.544735 8077 tablet_service.cc:1514] Processing DeleteTablet for tablet b9615087c4bf4b1bb4dfea49dbfec767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46300
I20260926 07:25:33.544735 8217 tablet_service.cc:1514] Processing DeleteTablet for tablet b9615087c4bf4b1bb4dfea49dbfec767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38044
I20260926 07:25:33.544902 8521 tablet_replica.cc:330] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4: stopping tablet replica
I20260926 07:25:33.545024 8521 raft_consensus.cc:2229] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:33.545157 8521 raft_consensus.cc:2258] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:33.545318 8519 tablet_replica.cc:330] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c: stopping tablet replica
I20260926 07:25:33.545435 8519 raft_consensus.cc:2229] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:33.545562 8519 raft_consensus.cc:2258] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:33.545863 8521 ts_tablet_manager.cc:1905] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:33.546351 8519 ts_tablet_manager.cc:1905] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:33.548007 8521 ts_tablet_manager.cc:1918] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:33.548061 8521 log.cc:1196] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/b9615087c4bf4b1bb4dfea49dbfec767
I20260926 07:25:33.548250 8521 ts_tablet_manager.cc:1939] T b9615087c4bf4b1bb4dfea49dbfec767 P 0d3989f13a8f4dc69973ebef84c00dd4: Deleting consensus metadata
I20260926 07:25:33.548703 8519 ts_tablet_manager.cc:1918] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:33.548925 7924 catalog_manager.cc:4841] TS 0d3989f13a8f4dc69973ebef84c00dd4 (127.0.98.2:37847): tablet b9615087c4bf4b1bb4dfea49dbfec767 (table test-table [id=48ad1fa2756942e6b379c6d76d777ebf]) successfully deleted
I20260926 07:25:33.549094 8356 tablet_service.cc:1514] Processing DeleteTablet for tablet b9615087c4bf4b1bb4dfea49dbfec767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43986
I20260926 07:25:33.549249 8520 tablet_replica.cc:330] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed: stopping tablet replica
I20260926 07:25:33.549345 8520 raft_consensus.cc:2229] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:25:33.549497 8520 raft_consensus.cc:2258] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:33.548780 8519 log.cc:1196] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-0/wal/wals/b9615087c4bf4b1bb4dfea49dbfec767
I20260926 07:25:33.549853 8519 ts_tablet_manager.cc:1939] T b9615087c4bf4b1bb4dfea49dbfec767 P 91b18845f2ad4985a6f64b755f73700c: Deleting consensus metadata
I20260926 07:25:33.550082 8520 ts_tablet_manager.cc:1905] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:33.550464 7926 catalog_manager.cc:4841] TS 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909): tablet b9615087c4bf4b1bb4dfea49dbfec767 (table test-table [id=48ad1fa2756942e6b379c6d76d777ebf]) successfully deleted
I20260926 07:25:33.551273 7940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:25:33.551654 7940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:33.552295 8520 ts_tablet_manager.cc:1918] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:33.552383 8520 log.cc:1196] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/b9615087c4bf4b1bb4dfea49dbfec767
I20260926 07:25:33.552627 8520 ts_tablet_manager.cc:1939] T b9615087c4bf4b1bb4dfea49dbfec767 P ac6edff32e5f48059f45b45dff9487ed: Deleting consensus metadata
I20260926 07:25:33.553408 7927 catalog_manager.cc:4841] TS ac6edff32e5f48059f45b45dff9487ed (127.0.98.3:44619): tablet b9615087c4bf4b1bb4dfea49dbfec767 (table test-table [id=48ad1fa2756942e6b379c6d76d777ebf]) successfully deleted
I20260926 07:25:33.555898 8077 tablet_service.cc:1467] Processing CreateTablet for tablet 8bffdc4d54894cfa8d5a8058ff483b82 (DEFAULT_TABLE table=test-table [id=c95953d95baf4b13aa084adae0637bef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:33.555976 8217 tablet_service.cc:1467] Processing CreateTablet for tablet 8bffdc4d54894cfa8d5a8058ff483b82 (DEFAULT_TABLE table=test-table [id=c95953d95baf4b13aa084adae0637bef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:33.556159 8077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bffdc4d54894cfa8d5a8058ff483b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:33.556244 8217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bffdc4d54894cfa8d5a8058ff483b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:33.557989 8452 tablet_bootstrap.cc:492] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c: Bootstrap starting.
I20260926 07:25:33.558657 8356 tablet_service.cc:1467] Processing CreateTablet for tablet 8bffdc4d54894cfa8d5a8058ff483b82 (DEFAULT_TABLE table=test-table [id=c95953d95baf4b13aa084adae0637bef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:33.558876 8356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bffdc4d54894cfa8d5a8058ff483b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:33.558893 8452 tablet_bootstrap.cc:654] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:33.559819 8451 tablet_bootstrap.cc:492] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4: Bootstrap starting.
I20260926 07:25:33.560529 8451 tablet_bootstrap.cc:654] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:33.561128 8453 tablet_bootstrap.cc:492] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed: Bootstrap starting.
I20260926 07:25:33.561328 8452 tablet_bootstrap.cc:492] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c: No bootstrap required, opened a new log
I20260926 07:25:33.561384 8452 ts_tablet_manager.cc:1397] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 07:25:33.561707 8452 raft_consensus.cc:348] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.561785 8452 raft_consensus.cc:374] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:33.561805 8452 raft_consensus.cc:729] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b18845f2ad4985a6f64b755f73700c, State: Initialized, Role: FOLLOWER
I20260926 07:25:33.561867 8453 tablet_bootstrap.cc:654] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:33.561933 8452 consensus_queue.cc:260] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.562211 8452 ts_tablet_manager.cc:1428] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:25:33.562928 8453 tablet_bootstrap.cc:492] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed: No bootstrap required, opened a new log
I20260926 07:25:33.562994 8453 ts_tablet_manager.cc:1397] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:33.563006 8451 tablet_bootstrap.cc:492] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4: No bootstrap required, opened a new log
I20260926 07:25:33.563119 8451 ts_tablet_manager.cc:1397] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 07:25:33.563359 8453 raft_consensus.cc:348] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.563409 8451 raft_consensus.cc:348] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.563462 8453 raft_consensus.cc:374] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:33.563484 8451 raft_consensus.cc:374] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:33.563489 8453 raft_consensus.cc:729] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac6edff32e5f48059f45b45dff9487ed, State: Initialized, Role: FOLLOWER
I20260926 07:25:33.563504 8451 raft_consensus.cc:729] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3989f13a8f4dc69973ebef84c00dd4, State: Initialized, Role: FOLLOWER
I20260926 07:25:33.563601 8453 consensus_queue.cc:260] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.563632 8451 consensus_queue.cc:260] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.563817 8453 ts_tablet_manager.cc:1428] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:25:33.563877 8451 ts_tablet_manager.cc:1428] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:25:33.894418 8457 raft_consensus.cc:482] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:33.894726 8457 raft_consensus.cc:504] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.895234 8457 leader_election.cc:288] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909), 0d3989f13a8f4dc69973ebef84c00dd4 (127.0.98.2:37847)
I20260926 07:25:33.895711 8237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bffdc4d54894cfa8d5a8058ff483b82" candidate_uuid: "ac6edff32e5f48059f45b45dff9487ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" is_pre_election: true
I20260926 07:25:33.895712 8097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bffdc4d54894cfa8d5a8058ff483b82" candidate_uuid: "ac6edff32e5f48059f45b45dff9487ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b18845f2ad4985a6f64b755f73700c" is_pre_election: true
I20260926 07:25:33.895869 8237 raft_consensus.cc:2454] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6edff32e5f48059f45b45dff9487ed in term 0.
I20260926 07:25:33.895887 8097 raft_consensus.cc:2454] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac6edff32e5f48059f45b45dff9487ed in term 0.
I20260926 07:25:33.896206 8308 leader_election.cc:302] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3989f13a8f4dc69973ebef84c00dd4, ac6edff32e5f48059f45b45dff9487ed; no voters:
I20260926 07:25:33.896382 8457 raft_consensus.cc:2781] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:33.896446 8457 raft_consensus.cc:482] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:33.896513 8457 raft_consensus.cc:3037] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:33.897462 8457 raft_consensus.cc:504] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.897838 8457 leader_election.cc:288] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [CANDIDATE]: Term 1 election: Requested vote from peers 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909), 0d3989f13a8f4dc69973ebef84c00dd4 (127.0.98.2:37847)
I20260926 07:25:33.898205 8097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bffdc4d54894cfa8d5a8058ff483b82" candidate_uuid: "ac6edff32e5f48059f45b45dff9487ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b18845f2ad4985a6f64b755f73700c"
I20260926 07:25:33.898206 8237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bffdc4d54894cfa8d5a8058ff483b82" candidate_uuid: "ac6edff32e5f48059f45b45dff9487ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3989f13a8f4dc69973ebef84c00dd4"
I20260926 07:25:33.898331 8097 raft_consensus.cc:3037] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:33.898331 8237 raft_consensus.cc:3037] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:33.899477 8097 raft_consensus.cc:2454] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6edff32e5f48059f45b45dff9487ed in term 1.
I20260926 07:25:33.899511 8237 raft_consensus.cc:2454] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac6edff32e5f48059f45b45dff9487ed in term 1.
I20260926 07:25:33.899766 8308 leader_election.cc:302] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3989f13a8f4dc69973ebef84c00dd4, ac6edff32e5f48059f45b45dff9487ed; no voters:
I20260926 07:25:33.899957 8457 raft_consensus.cc:2781] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:33.900066 8457 raft_consensus.cc:686] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [term 1 LEADER]: Becoming Leader. State: Replica: ac6edff32e5f48059f45b45dff9487ed, State: Running, Role: LEADER
I20260926 07:25:33.900207 8457 consensus_queue.cc:237] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } }
I20260926 07:25:33.902117 7939 catalog_manager.cc:5495] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed reported cstate change: term changed from 0 to 1, leader changed from <none> to ac6edff32e5f48059f45b45dff9487ed (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ac6edff32e5f48059f45b45dff9487ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac6edff32e5f48059f45b45dff9487ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44619 } health_report { overall_health: HEALTHY } } }
I20260926 07:25:33.939435 8097 raft_consensus.cc:1261] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Refusing update from remote peer ac6edff32e5f48059f45b45dff9487ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:33.939435 8237 raft_consensus.cc:1261] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Refusing update from remote peer ac6edff32e5f48059f45b45dff9487ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:33.939853 8457 consensus_queue.cc:1035] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3989f13a8f4dc69973ebef84c00dd4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:33.940037 8480 consensus_queue.cc:1035] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b18845f2ad4985a6f64b755f73700c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:33.970176 7939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:33.970306 7939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:33.972050 7939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b701b88e580c465199a1b51d30a77c7d: Sending DeleteTablet for 3 replicas of tablet 8bffdc4d54894cfa8d5a8058ff483b82
I20260926 07:25:33.972582 8077 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bffdc4d54894cfa8d5a8058ff483b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46300
I20260926 07:25:33.972582 8217 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bffdc4d54894cfa8d5a8058ff483b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38044
I20260926 07:25:33.972755 8519 tablet_replica.cc:330] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c: stopping tablet replica
I20260926 07:25:33.972723 8356 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bffdc4d54894cfa8d5a8058ff483b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43986
I20260926 07:25:33.972879 8519 raft_consensus.cc:2229] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:33.972908 8520 tablet_replica.cc:330] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed: stopping tablet replica
I20260926 07:25:33.973003 8519 raft_consensus.cc:2258] T 8bffdc4d54894cfa8d5a8058ff483b82 P 91b18845f2ad4985a6f64b755f73700c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:33.973070 8520 raft_consensus.cc:2229] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:25:33.973163 8521 tablet_replica.cc:330] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4: stopping tablet replica
I20260926 07:25:33.973239 8520 raft_consensus.cc:2258] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:33.973274 8521 raft_consensus.cc:2229] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:33.973364 8521 raft_consensus.cc:2258] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:33.973868 8520 ts_tablet_manager.cc:1905] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:33.974228 8521 ts_tablet_manager.cc:1905] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:33.974239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 8012
I20260926 07:25:33.976368 8520 ts_tablet_manager.cc:1918] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:33.976440 8520 log.cc:1196] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/8bffdc4d54894cfa8d5a8058ff483b82
I20260926 07:25:33.976614 8521 ts_tablet_manager.cc:1918] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:25:33.976681 8521 log.cc:1196] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/8bffdc4d54894cfa8d5a8058ff483b82
I20260926 07:25:33.976701 8520 ts_tablet_manager.cc:1939] T 8bffdc4d54894cfa8d5a8058ff483b82 P ac6edff32e5f48059f45b45dff9487ed: Deleting consensus metadata
I20260926 07:25:33.976924 8521 ts_tablet_manager.cc:1939] T 8bffdc4d54894cfa8d5a8058ff483b82 P 0d3989f13a8f4dc69973ebef84c00dd4: Deleting consensus metadata
I20260926 07:25:33.977355 7927 catalog_manager.cc:4841] TS ac6edff32e5f48059f45b45dff9487ed (127.0.98.3:44619): tablet 8bffdc4d54894cfa8d5a8058ff483b82 (table test-table [id=c95953d95baf4b13aa084adae0637bef]) successfully deleted
I20260926 07:25:33.977510 7924 catalog_manager.cc:4841] TS 0d3989f13a8f4dc69973ebef84c00dd4 (127.0.98.2:37847): tablet 8bffdc4d54894cfa8d5a8058ff483b82 (table test-table [id=c95953d95baf4b13aa084adae0637bef]) successfully deleted
W20260926 07:25:33.982573 7926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45909 (error 108)
I20260926 07:25:33.983345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 8152
I20260926 07:25:33.984088 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:33.983246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45909 (local address 127.0.0.1:46306)
0926 07:25:33.983891 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:33.983969 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
W20260926 07:25:33.984998 7926 catalog_manager.cc:4584] TS 91b18845f2ad4985a6f64b755f73700c (127.0.98.1:45909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bffdc4d54894cfa8d5a8058ff483b82: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45909: connect: Connection refused (error 111)
I20260926 07:25:33.990090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 8292
I20260926 07:25:33.996263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 7908
2026-09-26T07:25:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3424 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:25:34 dist-test-slave-g04g krb5kdc[8555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:25:34 dist-test-slave-g04g krb5kdc[8555](info): set up 2 sockets
Sep 26 07:25:34 dist-test-slave-g04g krb5kdc[8555](info): commencing operation
krb5kdc: starting...
W20260926 07:25:35.055773 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:25:35 dist-test-slave-g04g krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:25:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:25:35.178352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34903
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:35.285187 8568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:35.285441 8568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:35.285480 8568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:35.289482 8568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:35.289533 8568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:35.289552 8568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:35.289569 8568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:35.289613 8568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:35.294636 8568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:35.295585 8568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:35.296553 8568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:35.304405 8573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:35.304484 8574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:35.304602 8576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:35.304668 8568 server_base.cc:1034] running on GCE node
I20260926 07:25:35.305240 8568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:35.306289 8568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:35.307484 8568 hybrid_clock.cc:648] HybridClock initialized: now 1790407535307454 us; error 50 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T07:25:35Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1328 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:25:35 dist-test-slave-g04g krb5kdc[8581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:25:35 dist-test-slave-g04g krb5kdc[8581](info): set up 2 sockets
Sep 26 07:25:35 dist-test-slave-g04g krb5kdc[8581](info): commencing operation
krb5kdc: starting...
W20260926 07:25:36.359874 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:25:36 dist-test-slave-g04g krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:25:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:25:36.480556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38995
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:36.587893 8594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:36.588136 8594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:36.588176 8594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:36.592078 8594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:36.592129 8594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:36.592149 8594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:36.592164 8594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:36.592180 8594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:36.597256 8594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:36.598155 8594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:36.599131 8594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:36.607311 8602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:36.607303 8600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:36.607379 8594 server_base.cc:1034] running on GCE node
W20260926 07:25:36.607303 8599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:36.608053 8594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:36.609095 8594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:36.610252 8594 hybrid_clock.cc:648] HybridClock initialized: now 1790407536610241 us; error 30 us; skew 500 ppm
Sep 26 07:25:36 dist-test-slave-g04g krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:36.613072 8594 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:25:36.613900 8594 webserver.cc:460] Webserver started at http://127.0.98.62:45685/ using document root <none> and password file <none>
I20260926 07:25:36.614439 8594 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:36.614523 8594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:36.614744 8594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:36.616786 8594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "56f3c1c1ed75431cacb5abd0c876d8e9"
format_stamp: "Formatted at 2026-09-26 07:25:36 on dist-test-slave-g04g"
server_key: "d538dbf5ec01b6099a864f8037874a41"
server_key_iv: "a144edbed03ed96696e5370800435d1a"
server_key_version: "encryptionkey@0"
I20260926 07:25:36.617342 8594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "56f3c1c1ed75431cacb5abd0c876d8e9"
format_stamp: "Formatted at 2026-09-26 07:25:36 on dist-test-slave-g04g"
server_key: "d538dbf5ec01b6099a864f8037874a41"
server_key_iv: "a144edbed03ed96696e5370800435d1a"
server_key_version: "encryptionkey@0"
I20260926 07:25:36.621416 8594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:25:36.625483 8609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:36.626979 8594 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 07:25:36.627134 8594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "56f3c1c1ed75431cacb5abd0c876d8e9"
format_stamp: "Formatted at 2026-09-26 07:25:36 on dist-test-slave-g04g"
server_key: "d538dbf5ec01b6099a864f8037874a41"
server_key_iv: "a144edbed03ed96696e5370800435d1a"
server_key_version: "encryptionkey@0"
I20260926 07:25:36.627221 8594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:36.644860 8594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:36.648108 8594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:36.648262 8594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:36.655376 8594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38995
I20260926 07:25:36.655402 8661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38995 every 8 connection(s)
I20260926 07:25:36.656289 8594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:36.656607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 8594
I20260926 07:25:36.656749 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:36.657097 392 external_mini_cluster.cc:1428] Setting key ff12f1dfc62b9c23b0ac65aa1dad606b
I20260926 07:25:36.659183 8662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:36.665387 8662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9: Bootstrap starting.
Sep 26 07:25:36 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:36.667891 8662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:36.668828 8662 log.cc:824] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:36.670923 8662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9: No bootstrap required, opened a new log
I20260926 07:25:36.672856 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:36.658601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.0.98.62:38995)
0926 07:25:36.659114 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:36.659131 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:36.659168 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:25:36.659890 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:36.659907 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:36.660242 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:36.660540 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:36.661422 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:36.662486 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:36.663742 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:36.664237 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:36.664371 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:36.667512 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:36.667518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:36.667533 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:36.667564 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:36.669511 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:36.669965 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:36.669971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:36.669976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:36.670104 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:36.670416 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:36.670419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:36.670421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:36.670895 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:36.671110 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:36.671423 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 07:25:36.671701 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":159,"threads_started":1}
I20260926 07:25:36.674356 8662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38995 } }
I20260926 07:25:36.674553 8662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:36.674587 8662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56f3c1c1ed75431cacb5abd0c876d8e9, State: Initialized, Role: FOLLOWER
I20260926 07:25:36.675110 8662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38995 } }
I20260926 07:25:36.675243 8662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:36.675314 8662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:36.675415 8662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:36.676229 8662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38995 } }
I20260926 07:25:36.676566 8662 leader_election.cc:302] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56f3c1c1ed75431cacb5abd0c876d8e9; no voters:
I20260926 07:25:36.676848 8662 leader_election.cc:288] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:36.676996 8667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:36.677220 8667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 1 LEADER]: Becoming Leader. State: Replica: 56f3c1c1ed75431cacb5abd0c876d8e9, State: Running, Role: LEADER
I20260926 07:25:36.677512 8667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38995 } }
I20260926 07:25:36.678030 8662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:36.679386 8668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38995 } } }
I20260926 07:25:36.679509 8668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:36.679993 8669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56f3c1c1ed75431cacb5abd0c876d8e9. Latest consensus state: current_term: 1 leader_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38995 } } }
I20260926 07:25:36.680111 8669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:36.680613 8676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:36.683171 8676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:36.688238 8676 catalog_manager.cc:1347] Generated new cluster ID: 12e00c3351b54f5fb96c9116ac482ab0
I20260926 07:25:36.688300 8676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:36.709424 8676 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:36.710574 8676 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:36.725207 8676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9: Generated new TSK 0
I20260926 07:25:36.725996 8676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:25:36.758700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38995
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:36.867787 8688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:36.868036 8688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:36.868077 8688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:36.872139 8688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:36.872189 8688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:36.872287 8688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:36.877452 8688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:36.878569 8688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:36.879685 8688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:36.889155 8696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:36.889165 8694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:36.889165 8693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:36.889467 8688 server_base.cc:1034] running on GCE node
I20260926 07:25:36.889894 8688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:36.890558 8688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:36.891736 8688 hybrid_clock.cc:648] HybridClock initialized: now 1790407536891720 us; error 29 us; skew 500 ppm
Sep 26 07:25:36 dist-test-slave-g04g krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:36.894295 8688 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:25:36.895170 8688 webserver.cc:460] Webserver started at http://127.0.98.1:37371/ using document root <none> and password file <none>
I20260926 07:25:36.895704 8688 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:36.895790 8688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:36.895996 8688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:36.897943 8688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "c6d1fc45db7d46ac90e0990da38f90b7"
format_stamp: "Formatted at 2026-09-26 07:25:36 on dist-test-slave-g04g"
server_key: "eb907f5d51a70993354cb968e2960599"
server_key_iv: "6a1b76b7f1e23aee018bc152f2019f2a"
server_key_version: "encryptionkey@0"
I20260926 07:25:36.898494 8688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c6d1fc45db7d46ac90e0990da38f90b7"
format_stamp: "Formatted at 2026-09-26 07:25:36 on dist-test-slave-g04g"
server_key: "eb907f5d51a70993354cb968e2960599"
server_key_iv: "6a1b76b7f1e23aee018bc152f2019f2a"
server_key_version: "encryptionkey@0"
I20260926 07:25:36.902040 8688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 07:25:36.904412 8703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:36.905591 8688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:36.905706 8688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "c6d1fc45db7d46ac90e0990da38f90b7"
format_stamp: "Formatted at 2026-09-26 07:25:36 on dist-test-slave-g04g"
server_key: "eb907f5d51a70993354cb968e2960599"
server_key_iv: "6a1b76b7f1e23aee018bc152f2019f2a"
server_key_version: "encryptionkey@0"
I20260926 07:25:36.905812 8688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:36.925017 8688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:36.927768 8688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:36.927960 8688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:36.928499 8688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:36.929267 8688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:36.929312 8688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:36.929350 8688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:36.929404 8688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:36.938023 8688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42949
I20260926 07:25:36.938048 8816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42949 every 8 connection(s)
I20260926 07:25:36.938858 8688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:36.944609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 8688
I20260926 07:25:36.944837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:25:36.945221 392 external_mini_cluster.cc:1428] Setting key c1ba55777b8d23b91f669342c8bc2fb3
Sep 26 07:25:36 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:36.954488 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:36.940871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37673 (local address 127.0.98.62:38995)
0926 07:25:36.941004 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:36.941008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:36.941852 (+ 844us) server_negotiation.cc:407] Connection header received
0926 07:25:36.943272 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:36.943276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:36.943335 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:36.943418 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:36.944655 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:36.945421 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:36.946993 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:36.947276 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:36.947371 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:36.950851 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:36.950854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:36.950856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:36.950880 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:36.952414 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:36.952944 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:36.952949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:36.952951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:36.953031 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:36.953379 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:36.953382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:36.953384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:36.953562 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:36.953666 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:36.954167 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 07:25:36.954313 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 07:25:36.955402 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:36.941257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:37673)
0926 07:25:36.941698 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:36.941740 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:36.943016 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:36.943576 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:36.943582 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:36.943931 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:36.944472 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:36.944493 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:36.945564 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:36.945568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:36.946827 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:36.946837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:36.947451 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:36.947459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:36.947717 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:36.948446 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:36.948474 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:36.950614 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:36.952535 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:36.952541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:36.952557 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:36.952837 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:36.953124 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:36.953128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:36.953130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:36.953277 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:36.953664 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:36.953671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:36.953929 (+ 258us) client_negotiation.cc:769] Sending connection context
0926 07:25:36.954119 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 07:25:36.954448 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20260926 07:25:36.956600 8817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38995
I20260926 07:25:36.956838 8817 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:36.957397 8817 heartbeater.cc:511] Master 127.0.98.62:38995 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 07:25:36.959158 8626 ts_manager.cc:194] Registered new tserver with Master: c6d1fc45db7d46ac90e0990da38f90b7 (127.0.98.1:42949)
I20260926 07:25:36.960297 8626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:25:36.975123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38995
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:37.085798 8822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:37.086061 8822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:37.086141 8822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:37.090248 8822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:37.090330 8822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:37.090415 8822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:37.095656 8822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:37.096644 8822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:37.097630 8822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:37.106936 8828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:37.107112 8822 server_base.cc:1034] running on GCE node
W20260926 07:25:37.107009 8830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:37.106937 8827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:37.107622 8822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:37.108284 8822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:37.109498 8822 hybrid_clock.cc:648] HybridClock initialized: now 1790407537109482 us; error 28 us; skew 500 ppm
Sep 26 07:25:37 dist-test-slave-g04g krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:37.112255 8822 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:25:37.113042 8822 webserver.cc:460] Webserver started at http://127.0.98.2:34591/ using document root <none> and password file <none>
I20260926 07:25:37.113636 8822 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:37.113725 8822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:37.113932 8822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:37.115952 8822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "317f25fb26584eecbae3f644c9a1df48"
format_stamp: "Formatted at 2026-09-26 07:25:37 on dist-test-slave-g04g"
server_key: "694ada4d8e6b84d3e9aa8ab7921dcd16"
server_key_iv: "b0c880b57fc4b7209f741d157dfb49ff"
server_key_version: "encryptionkey@0"
I20260926 07:25:37.116503 8822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "317f25fb26584eecbae3f644c9a1df48"
format_stamp: "Formatted at 2026-09-26 07:25:37 on dist-test-slave-g04g"
server_key: "694ada4d8e6b84d3e9aa8ab7921dcd16"
server_key_iv: "b0c880b57fc4b7209f741d157dfb49ff"
server_key_version: "encryptionkey@0"
I20260926 07:25:37.119972 8822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:37.122275 8837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:37.123447 8822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:37.123581 8822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "317f25fb26584eecbae3f644c9a1df48"
format_stamp: "Formatted at 2026-09-26 07:25:37 on dist-test-slave-g04g"
server_key: "694ada4d8e6b84d3e9aa8ab7921dcd16"
server_key_iv: "b0c880b57fc4b7209f741d157dfb49ff"
server_key_version: "encryptionkey@0"
I20260926 07:25:37.123685 8822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:37.146697 8822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:37.149657 8822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:37.149856 8822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:37.150418 8822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:37.151347 8822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:37.151422 8822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:37.151492 8822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:37.151544 8822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:37.160995 8822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44587
I20260926 07:25:37.161021 8950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44587 every 8 connection(s)
I20260926 07:25:37.161813 8822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:25:37 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:37.171200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 8822
I20260926 07:25:37.171360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:25:37.171669 392 external_mini_cluster.cc:1428] Setting key 4360f067a441aef9c380a09db837e73c
I20260926 07:25:37.175407 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:37.163492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50259 (local address 127.0.98.62:38995)
0926 07:25:37.163618 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:37.163622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:37.164520 (+ 898us) server_negotiation.cc:407] Connection header received
0926 07:25:37.165828 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:37.165831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:37.165876 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:37.165950 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:37.166986 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.167569 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.168641 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.168845 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.168916 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:37.171482 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:37.171487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:37.171489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:37.171516 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:37.173160 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:37.173722 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:37.173725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:37.173726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:37.173799 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:37.174177 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:37.174179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:37.174180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:37.174338 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:37.174416 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:37.175002 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 07:25:37.175178 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:25:37.176247 8953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:37.163792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:50259)
0926 07:25:37.164360 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:37.164400 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:37.165626 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:37.166069 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:37.166076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:37.166399 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:37.166865 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:37.166878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.167674 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.167677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:37.168543 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:37.168551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.168962 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.168965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:37.169098 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:37.169670 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:37.169693 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:37.171265 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:37.173282 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:37.173287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:37.173301 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:37.173623 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:37.173890 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:37.173896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:37.173901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:37.174083 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:37.174433 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:37.174442 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:37.174760 (+ 318us) client_negotiation.cc:769] Sending connection context
0926 07:25:37.174971 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 07:25:37.175317 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":274,"threads_started":1}
I20260926 07:25:37.177441 8951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38995
I20260926 07:25:37.177671 8951 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:37.178200 8951 heartbeater.cc:511] Master 127.0.98.62:38995 requested a full tablet report, sending...
I20260926 07:25:37.179267 8626 ts_manager.cc:194] Registered new tserver with Master: 317f25fb26584eecbae3f644c9a1df48 (127.0.98.2:44587)
I20260926 07:25:37.179893 8626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50259
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:25:37.198402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38995
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:37.305881 8956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:37.306165 8956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:37.306243 8956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:37.310902 8956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:37.310979 8956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:37.311100 8956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:37.316298 8956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:37.317301 8956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:37.318295 8956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:37.327378 8964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:37.327426 8962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:37.327426 8961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:37.327598 8956 server_base.cc:1034] running on GCE node
I20260926 07:25:37.327978 8956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:37.328590 8956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:37.329780 8956 hybrid_clock.cc:648] HybridClock initialized: now 1790407537329738 us; error 52 us; skew 500 ppm
Sep 26 07:25:37 dist-test-slave-g04g krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:37.332564 8956 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:25:37.333444 8956 webserver.cc:460] Webserver started at http://127.0.98.3:34373/ using document root <none> and password file <none>
I20260926 07:25:37.333978 8956 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:37.334065 8956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:37.334311 8956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:37.336555 8956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "82d4d708f22b4e7b9211ded840b519c1"
format_stamp: "Formatted at 2026-09-26 07:25:37 on dist-test-slave-g04g"
server_key: "d71c651361789914cd819194f09da3d3"
server_key_iv: "5a9a7f699aea1ec765da1c3afe87288b"
server_key_version: "encryptionkey@0"
I20260926 07:25:37.337138 8956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "82d4d708f22b4e7b9211ded840b519c1"
format_stamp: "Formatted at 2026-09-26 07:25:37 on dist-test-slave-g04g"
server_key: "d71c651361789914cd819194f09da3d3"
server_key_iv: "5a9a7f699aea1ec765da1c3afe87288b"
server_key_version: "encryptionkey@0"
I20260926 07:25:37.340685 8956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:25:37.343143 8971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:37.344379 8956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:37.344515 8956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "82d4d708f22b4e7b9211ded840b519c1"
format_stamp: "Formatted at 2026-09-26 07:25:37 on dist-test-slave-g04g"
server_key: "d71c651361789914cd819194f09da3d3"
server_key_iv: "5a9a7f699aea1ec765da1c3afe87288b"
server_key_version: "encryptionkey@0"
I20260926 07:25:37.344625 8956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:37.370241 8956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:37.373689 8956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:37.373891 8956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:37.374473 8956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:37.375349 8956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:37.375420 8956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:37.375494 8956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:37.375540 8956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:37.384637 8956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43789
I20260926 07:25:37.384662 9084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43789 every 8 connection(s)
I20260926 07:25:37.385485 8956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:37.394486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 8956
I20260926 07:25:37.394621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:37.394912 392 external_mini_cluster.cc:1428] Setting key fd364f394b52b33ee7abbbbedab789f9
Sep 26 07:25:37 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:37.398929 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:37.387285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48833 (local address 127.0.98.62:38995)
0926 07:25:37.387420 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:37.387424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:37.388300 (+ 876us) server_negotiation.cc:407] Connection header received
0926 07:25:37.389666 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:37.389669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:37.389722 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:37.389832 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:37.390944 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.391517 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.392681 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.392902 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.393009 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:37.395567 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:37.395571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:37.395577 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:37.395610 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:37.397244 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:37.397700 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:37.397702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:37.397704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:37.397768 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:37.398076 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:37.398078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:37.398079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:37.398227 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:37.398297 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:37.398715 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 07:25:37.398802 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:25:37.399909 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:37.387743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:48833)
0926 07:25:37.388153 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:37.388192 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:37.389461 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:37.390004 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:37.390011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:37.390344 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:37.390791 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:37.390804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.391639 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.391643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:37.392561 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:37.392569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.393052 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.393056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:37.393213 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:37.393744 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:37.393759 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:37.395419 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:37.397360 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:37.397365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:37.397375 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:37.397615 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:37.397873 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:37.397875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:37.397876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:37.397999 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:37.398312 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:37.398317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:37.398526 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 07:25:37.398673 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 07:25:37.398939 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
I20260926 07:25:37.400997 9085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38995
I20260926 07:25:37.401222 9085 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:37.401665 9085 heartbeater.cc:511] Master 127.0.98.62:38995 requested a full tablet report, sending...
I20260926 07:25:37.402576 8626 ts_manager.cc:194] Registered new tserver with Master: 82d4d708f22b4e7b9211ded840b519c1 (127.0.98.3:43789)
I20260926 07:25:37.403189 8626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48833
I20260926 07:25:37.408612 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 07:25:37.482405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 8688
I20260926 07:25:37.487833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 8822
I20260926 07:25:37.492967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 8956
I20260926 07:25:37.498226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 8594
I20260926 07:25:37.504254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38995
--webserver_interface=127.0.98.62
--webserver_port=45685
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:37.612666 9094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:37.612926 9094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:37.612993 9094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:37.616963 9094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:37.617015 9094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:37.617034 9094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:37.617092 9094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:37.617115 9094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:37.622359 9094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:37.623377 9094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:37.624377 9094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:37.632756 9102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:37.632854 9100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:37.632777 9099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:37.633034 9094 server_base.cc:1034] running on GCE node
I20260926 07:25:37.633576 9094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:37.634684 9094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:37.635890 9094 hybrid_clock.cc:648] HybridClock initialized: now 1790407537635863 us; error 32 us; skew 500 ppm
Sep 26 07:25:37 dist-test-slave-g04g krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:37.638590 9094 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:25:37.639554 9094 webserver.cc:460] Webserver started at http://127.0.98.62:45685/ using document root <none> and password file <none>
I20260926 07:25:37.640072 9094 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:37.640139 9094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:37.643707 9094 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:37.645663 9109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:37.646852 9094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:37.646960 9094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "56f3c1c1ed75431cacb5abd0c876d8e9"
format_stamp: "Formatted at 2026-09-26 07:25:36 on dist-test-slave-g04g"
server_key: "d538dbf5ec01b6099a864f8037874a41"
server_key_iv: "a144edbed03ed96696e5370800435d1a"
server_key_version: "encryptionkey@0"
I20260926 07:25:37.647516 9094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:37.669978 9094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:37.673337 9094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:37.673491 9094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:37.680861 9094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38995
I20260926 07:25:37.680889 9161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38995 every 8 connection(s)
I20260926 07:25:37.681978 9094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:37.686432 9162 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:25:37.688177 9162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9: Bootstrap starting.
I20260926 07:25:37.690322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 9094
I20260926 07:25:37.691077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42949
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38995
--webserver_port=37371
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:25:37.705890 9162 log.cc:824] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:37.712779 9162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:25:37.713188 9162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9: Bootstrap complete.
I20260926 07:25:37.717368 9162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38995 } }
I20260926 07:25:37.717747 9162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56f3c1c1ed75431cacb5abd0c876d8e9, State: Initialized, Role: FOLLOWER
I20260926 07:25:37.718248 9162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38995 } }
I20260926 07:25:37.718403 9162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:37.718477 9162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:37.718582 9162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:25:37.720516 9162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38995 } }
I20260926 07:25:37.720850 9162 leader_election.cc:302] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56f3c1c1ed75431cacb5abd0c876d8e9; no voters:
I20260926 07:25:37.721186 9162 leader_election.cc:288] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:25:37.721273 9166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:25:37.721472 9166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [term 2 LEADER]: Becoming Leader. State: Replica: 56f3c1c1ed75431cacb5abd0c876d8e9, State: Running, Role: LEADER
I20260926 07:25:37.721776 9166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38995 } }
I20260926 07:25:37.722388 9162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:37.722714 9168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56f3c1c1ed75431cacb5abd0c876d8e9. Latest consensus state: current_term: 2 leader_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38995 } } }
I20260926 07:25:37.722844 9168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:37.723097 9167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f3c1c1ed75431cacb5abd0c876d8e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38995 } } }
I20260926 07:25:37.723217 9167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:37.723538 9170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:37.726792 9170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:37.727607 9170 catalog_manager.cc:1259] Loaded cluster ID: 12e00c3351b54f5fb96c9116ac482ab0
I20260926 07:25:37.727692 9170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:37.729969 9170 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:37.730638 9170 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 56f3c1c1ed75431cacb5abd0c876d8e9: Loaded TSK: 0
I20260926 07:25:37.731837 9170 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:25:37.802465 9164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:37.802731 9164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:37.802819 9164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:37.806851 9164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:37.806928 9164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:37.807016 9164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:37.812178 9164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37371
--tserver_master_addrs=127.0.98.62:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:37.813127 9164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:37.814103 9164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:37.824633 9189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:37.824685 9188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:37.824641 9191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:37.825314 9164 server_base.cc:1034] running on GCE node
I20260926 07:25:37.825727 9164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:37.826396 9164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:37.827564 9164 hybrid_clock.cc:648] HybridClock initialized: now 1790407537827534 us; error 45 us; skew 500 ppm
Sep 26 07:25:37 dist-test-slave-g04g krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407537, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:37.830292 9164 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 07:25:37.831266 9164 webserver.cc:460] Webserver started at http://127.0.98.1:37371/ using document root <none> and password file <none>
I20260926 07:25:37.831836 9164 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:37.831909 9164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:37.835657 9164 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 07:25:37.837757 9198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:37.838973 9164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 07:25:37.839126 9164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "c6d1fc45db7d46ac90e0990da38f90b7"
format_stamp: "Formatted at 2026-09-26 07:25:36 on dist-test-slave-g04g"
server_key: "eb907f5d51a70993354cb968e2960599"
server_key_iv: "6a1b76b7f1e23aee018bc152f2019f2a"
server_key_version: "encryptionkey@0"
I20260926 07:25:37.839643 9164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:37.856532 9164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:37.859943 9164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:37.860144 9164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:37.860805 9164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:37.861624 9164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:37.861670 9164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:37.861733 9164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:37.861764 9164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:37.872310 9164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42949
I20260926 07:25:37.872335 9311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42949 every 8 connection(s)
I20260926 07:25:37.873287 9164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:37.877595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 9164
I20260926 07:25:37.878082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44587
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38995
--webserver_port=34591
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:25:37 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:37 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:37.887809 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:37.875447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:58951)
0926 07:25:37.875875 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:37.875925 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:37.877183 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:37.878482 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:37.878489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:37.878862 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:37.879510 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:37.879532 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.880778 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.880785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:37.882327 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:37.882342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.883091 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.883097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:37.883340 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:37.884024 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:37.884050 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:37.886869 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":131,"threads_started":1}
W20260926 07:25:37.887799 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:37.875283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58951 (local address 127.0.98.62:38995)
0926 07:25:37.875693 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:37.875708 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:37.876037 (+ 329us) server_negotiation.cc:407] Connection header received
0926 07:25:37.877595 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:37.877612 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:37.878010 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:37.878381 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:37.879695 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.880631 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.882463 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.882902 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.883021 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:37.887104 (+ 4083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58951: BlockingRecv error: recv got EOF from 127.0.98.1:58951 (error 108)
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
W20260926 07:25:37.888235 9312 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38995 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38995: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 07:25:37 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:37 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:37.893581 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:37.888702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:36305)
0926 07:25:37.888808 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:37.888824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:37.888901 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:37.889142 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:37.889145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:37.889294 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:37.889383 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:37.889387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.890104 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.890107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:37.890639 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:37.890646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.891129 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.891132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:37.891225 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:37.891694 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:37.891705 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:37.893448 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 07:25:37.893674 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:37.888780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36305 (local address 127.0.98.62:38995)
0926 07:25:37.888889 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:37.888893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:37.888906 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:25:37.888994 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:37.888997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:37.889043 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:37.889129 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:37.889491 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.889991 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.890753 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.890982 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.891128 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:37.893561 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36305: BlockingRecv error: recv got EOF from 127.0.98.1:36305 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 07:25:37 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:37 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:37.898720 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:37.894090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:32789)
0926 07:25:37.894177 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:37.894191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:37.894284 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:37.894516 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:37.894519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:37.894657 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:37.894717 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:37.894721 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.895452 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.895459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:37.895960 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:37.895967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.896393 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.896396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:37.896470 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:37.896917 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:37.896930 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:37.898592 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 07:25:37.898805 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:37.894188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32789 (local address 127.0.98.62:38995)
0926 07:25:37.894314 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:37.894319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:37.894333 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:37.894370 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:37.894376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:37.894421 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:37.894503 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:37.894821 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.895355 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.896072 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:37.896283 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:37.896393 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:37.898693 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32789: BlockingRecv error: recv got EOF from 127.0.98.1:32789 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 07:25:37.899052 9312 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:25:37.995818 9316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:37.996066 9316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:37.996107 9316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:38.000237 9316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:38.000310 9316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:38.000452 9316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:38.006094 9316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34591
--tserver_master_addrs=127.0.98.62:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:38.007305 9316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:38.008401 9316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:38.018100 9322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:38.018111 9324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:38.018111 9321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:38.018388 9316 server_base.cc:1034] running on GCE node
I20260926 07:25:38.018770 9316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:38.019554 9316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:38.020735 9316 hybrid_clock.cc:648] HybridClock initialized: now 1790407538020704 us; error 44 us; skew 500 ppm
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407538, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:38.023582 9316 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 07:25:38.024540 9316 webserver.cc:460] Webserver started at http://127.0.98.2:34591/ using document root <none> and password file <none>
I20260926 07:25:38.025115 9316 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:38.025209 9316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:38.028997 9316 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 07:25:38.030980 9331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:38.032274 9316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:38.032410 9316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "317f25fb26584eecbae3f644c9a1df48"
format_stamp: "Formatted at 2026-09-26 07:25:37 on dist-test-slave-g04g"
server_key: "694ada4d8e6b84d3e9aa8ab7921dcd16"
server_key_iv: "b0c880b57fc4b7209f741d157dfb49ff"
server_key_version: "encryptionkey@0"
I20260926 07:25:38.032945 9316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:38.045852 9316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:38.048645 9316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:38.048843 9316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:38.049388 9316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:38.050155 9316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:38.050225 9316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:38.050293 9316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:38.050328 9316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:38.059300 9316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44587
I20260926 07:25:38.059341 9444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44587 every 8 connection(s)
I20260926 07:25:38.060256 9316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:25:38.064337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 9316
I20260926 07:25:38.064790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43789
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38995
--webserver_port=34373
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:25:38.072549 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:38.061933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52247 (local address 127.0.98.62:38995)
0926 07:25:38.062098 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:38.062103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:38.062902 (+ 799us) server_negotiation.cc:407] Connection header received
0926 07:25:38.064292 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:38.064297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:38.064362 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:38.064466 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:38.065717 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.066388 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.067994 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.068212 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.068294 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.072449 (+ 4155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52247: BlockingRecv error: recv got EOF from 127.0.98.2:52247 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:25:38.073432 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:38.062332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:52247)
0926 07:25:38.062746 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:38.062786 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:38.064073 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:38.064651 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:38.064657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:38.065093 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:38.065565 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.065579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.066520 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.066524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.067851 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.067866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.068378 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.068383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.068609 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.069244 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:38.069272 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:38.072326 (+ 3054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
W20260926 07:25:38.073894 9445 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38995 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38995: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:38.079234 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:38.074378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:56095)
0926 07:25:38.074471 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:38.074486 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:38.074583 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:38.074828 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:38.074831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:38.074981 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:38.075129 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.075135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.075839 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.075845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.076387 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.076394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.076834 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.076838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.076926 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.077409 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:38.077423 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:38.079121 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 07:25:38.079471 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:38.074478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56095 (local address 127.0.98.62:38995)
0926 07:25:38.074605 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:38.074612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:38.074628 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:25:38.074671 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:38.074674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:38.074724 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:38.074814 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:38.075240 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.075734 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.076501 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.076726 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.076835 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.079244 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56095: BlockingRecv error: recv got EOF from 127.0.98.2:56095 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:38.084769 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:38.079808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:56187)
0926 07:25:38.079894 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:38.079911 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:38.079972 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:38.080176 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:38.080178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:38.080291 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:38.080353 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.080357 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.081301 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.081303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.081809 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.081817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.082357 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.082360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.082413 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.082858 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:38.082869 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:38.084686 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 07:25:38.084915 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:38.079871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56187 (local address 127.0.98.62:38995)
0926 07:25:38.079972 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:38.079976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:38.079990 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:38.080025 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:38.080028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:38.080067 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:38.080149 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:38.080446 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.081199 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.081917 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.082254 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.082362 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.084811 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56187: BlockingRecv error: recv got EOF from 127.0.98.2:56187 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 07:25:38.085244 9445 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:25:38.179872 9448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:38.180150 9448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:38.180231 9448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:38.184226 9448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:38.184276 9448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:38.184332 9448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:38.189306 9448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34373
--tserver_master_addrs=127.0.98.62:38995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:38.190181 9448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:38.191265 9448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:38.200351 9453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:38.200359 9456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:38.200366 9454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:38.200408 9448 server_base.cc:1034] running on GCE node
I20260926 07:25:38.200913 9448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:38.201545 9448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:38.202735 9448 hybrid_clock.cc:648] HybridClock initialized: now 1790407538202706 us; error 42 us; skew 500 ppm
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407538, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:38.205569 9448 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 07:25:38.206378 9448 webserver.cc:460] Webserver started at http://127.0.98.3:34373/ using document root <none> and password file <none>
I20260926 07:25:38.206852 9448 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:38.206920 9448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:38.210448 9448 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:38.212553 9463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:38.213793 9448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:38.214031 9448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "82d4d708f22b4e7b9211ded840b519c1"
format_stamp: "Formatted at 2026-09-26 07:25:37 on dist-test-slave-g04g"
server_key: "d71c651361789914cd819194f09da3d3"
server_key_iv: "5a9a7f699aea1ec765da1c3afe87288b"
server_key_version: "encryptionkey@0"
I20260926 07:25:38.214542 9448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:38.225224 9448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:38.228307 9448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:38.228488 9448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:38.229015 9448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:38.229876 9448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:38.229943 9448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:38.230011 9448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:38.230062 9448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:38.239285 9448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43789
I20260926 07:25:38.239339 9576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43789 every 8 connection(s)
I20260926 07:25:38.240291 9448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:38.241174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 9448
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:38.251398 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:38.241901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.0.98.62:38995)
0926 07:25:38.242048 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:38.242053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:38.242294 (+ 241us) server_negotiation.cc:407] Connection header received
0926 07:25:38.242402 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:38.242405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:38.242453 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:38.242545 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:38.244013 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.244550 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.245423 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.245753 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.245863 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.247366 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:38.247372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:38.247383 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:38.247407 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:38.249122 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:38.249563 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:38.249568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:38.249577 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:38.249726 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:38.250058 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:38.250062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:38.250066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:38.250414 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:38.250609 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:38.251001 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 07:25:38.251273 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:25:38.252719 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:38.243558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34523 (local address 127.0.98.62:38995)
0926 07:25:38.244205 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:38.244208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:38.244220 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:25:38.245766 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:38.245769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:38.245835 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:38.245924 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:38.247221 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.247775 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.248879 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.249101 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.249213 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.252610 (+ 3397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34523: BlockingRecv error: recv got EOF from 127.0.98.3:34523 (error 108)
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":99,"threads_started":1}
I20260926 07:25:38.253281 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:38.242662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:34523)
0926 07:25:38.243259 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:38.243351 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:38.245558 (+ 2207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:38.246080 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:38.246086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:38.246568 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:38.247014 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.247078 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.247900 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.247904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.248758 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.248767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.249247 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.249252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.249406 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.249892 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:38.249911 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:38.252489 (+ 2578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":138,"threads_started":1}
W20260926 07:25:38.253680 9577 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38995 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38995: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:38.258553 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:38.254112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:55637)
0926 07:25:38.254224 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:38.254238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:38.254308 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:38.254522 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:38.254524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:38.254647 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:38.254728 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.254732 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.255465 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.255467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.256032 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.256040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.256439 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.256442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.256512 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.256898 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:38.256909 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:38.258475 (+ 1566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 07:25:38.258714 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:38.254177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55637 (local address 127.0.98.62:38995)
0926 07:25:38.254283 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:38.254287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:38.254319 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:25:38.254395 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:38.254398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:38.254436 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:38.254506 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:38.254819 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.255363 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.256129 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.256330 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.256399 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.258581 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55637: BlockingRecv error: recv got EOF from 127.0.98.3:55637 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:38.263212 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:38.258996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:42877)
0926 07:25:38.259133 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:38.259146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:38.259209 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:38.259392 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:38.259394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:38.259504 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:38.259564 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.259568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.260177 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.260179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.260684 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.260691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.261079 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.261082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.261142 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.261500 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:38.261511 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:38.263131 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":58}
W20260926 07:25:38.263463 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:38.259084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42877 (local address 127.0.98.62:38995)
0926 07:25:38.259185 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:38.259188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:38.259199 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:25:38.259271 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:38.259273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:38.259303 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:38.259364 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:38.259650 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.260094 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.260775 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.260972 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.261042 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.263367 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42877: BlockingRecv error: recv got EOF from 127.0.98.3:42877 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 07:25:38.263471 9577 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:38 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:38.905716 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:38.899676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:59767)
0926 07:25:38.899959 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:38.899976 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:38.900106 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:38.900393 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:38.900396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:38.900640 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:38.900850 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.900862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.901834 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.901836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.902585 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:38.902594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.903014 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.903016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:38.903157 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.903684 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:38.903704 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:38.905542 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
W20260926 07:25:38.905786 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:38.899755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59767 (local address 127.0.98.62:38995)
0926 07:25:38.900040 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:38.900044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:38.900058 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:38.900191 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:38.900194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:38.900268 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:38.900374 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:38.900957 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.901734 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.902698 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:38.902913 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:38.903000 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:38.905685 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59767: BlockingRecv error: recv got EOF from 127.0.98.1:59767 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":117,"threads_started":1}
Sep 26 07:25:39 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:39 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:39.091013 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:39.085873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:40909)
0926 07:25:39.086129 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:39.086148 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:39.086243 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:39.086473 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:39.086475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:39.086648 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:39.086754 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:39.086759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:39.087560 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:39.087562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:39.088170 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:39.088179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:39.088566 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:39.088568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:39.088641 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:39.089130 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:39.089149 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:39.090895 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
W20260926 07:25:39.091202 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:39.085954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40909 (local address 127.0.98.62:38995)
0926 07:25:39.086089 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:39.086098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:39.086214 (+ 116us) server_negotiation.cc:407] Connection header received
0926 07:25:39.086332 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:39.086334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:39.086377 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:39.086438 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:39.086849 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:39.087458 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:39.088272 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:39.088474 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:39.088546 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:39.091073 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40909: BlockingRecv error: recv got EOF from 127.0.98.2:40909 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 07:25:39 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:39 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:39.269295 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:39.264078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:33719)
0926 07:25:39.264365 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:39.264378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:39.264470 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:39.264699 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:39.264701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:39.264879 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:39.264986 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:39.264991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:39.265674 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:39.265676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:39.266231 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:39.266258 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:39.266640 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:39.266643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:39.266711 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:39.267227 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:39.267249 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:39.269114 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":100,"threads_started":1}
W20260926 07:25:39.269336 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:39.264172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33719 (local address 127.0.98.62:38995)
0926 07:25:39.264328 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:39.264332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:39.264439 (+ 107us) server_negotiation.cc:407] Connection header received
0926 07:25:39.264555 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:39.264558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:39.264600 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:39.264671 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:39.265077 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:39.265581 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:39.266344 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:39.266547 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:39.266619 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:39.269235 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33719: BlockingRecv error: recv got EOF from 127.0.98.3:33719 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 07:25:39 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:39 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:39.911844 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:39.906544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:36327)
0926 07:25:39.906831 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:39.906845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:39.906932 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:39.907177 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:39.907179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:39.907350 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:39.907445 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:39.907450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:39.908198 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:39.908202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:39.908783 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:39.908792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:39.909201 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:39.909204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:39.909267 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:39.909795 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:39.909807 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:39.911707 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":119,"threads_started":1}
W20260926 07:25:39.912043 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:39.906630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36327 (local address 127.0.98.62:38995)
0926 07:25:39.906888 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:39.906892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:39.906905 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:25:39.906999 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:39.907001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:39.907061 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:39.907137 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:39.907543 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:39.908066 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:39.908892 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:39.909099 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:39.909176 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:39.911853 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36327: BlockingRecv error: recv got EOF from 127.0.98.1:36327 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":109,"threads_started":1}
Sep 26 07:25:40 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:40 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:40.096963 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:40.091819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:54235)
0926 07:25:40.092095 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:40.092108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:40.092203 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:40.092409 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:40.092411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:40.092579 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:40.092669 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:40.092674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:40.093424 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:40.093427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:40.094017 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:40.094027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:40.094414 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:40.094416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:40.094497 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:40.094958 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:40.094970 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:40.096831 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
W20260926 07:25:40.097074 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:40.091903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54235 (local address 127.0.98.62:38995)
0926 07:25:40.092034 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:40.092038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:40.092177 (+ 139us) server_negotiation.cc:407] Connection header received
0926 07:25:40.092279 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:40.092281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:40.092322 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:40.092381 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:40.092760 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:40.093265 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:40.094120 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:40.094323 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:40.094394 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:40.096956 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54235: BlockingRecv error: recv got EOF from 127.0.98.2:54235 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 07:25:40 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:40 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:40.275815 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:40.270164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:46385)
0926 07:25:40.270480 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:40.270495 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:40.270654 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:40.270980 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:40.270983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:40.271275 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:40.271440 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:40.271453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:40.272216 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:40.272219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:40.272830 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:40.272840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:40.273245 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:40.273248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:40.273311 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:40.273843 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:40.273854 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:40.275677 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":93,"threads_started":1}
W20260926 07:25:40.275933 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:40.270355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46385 (local address 127.0.98.62:38995)
0926 07:25:40.270479 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:40.270485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:40.270620 (+ 135us) server_negotiation.cc:407] Connection header received
0926 07:25:40.270768 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:40.270771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:40.270839 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:40.270948 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:40.271566 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:40.272093 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:40.272938 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:40.273147 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:40.273220 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:40.275817 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46385: BlockingRecv error: recv got EOF from 127.0.98.3:46385 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 07:25:40 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:40 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:40.922614 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:40.912653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:43641)
0926 07:25:40.912955 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:40.912969 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:40.913096 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:40.913358 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:40.913362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:40.913562 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:40.913667 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:40.913673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:40.914414 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:40.914417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:40.915014 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:40.915024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:40.915497 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:40.915500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:40.915565 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:40.920613 (+ 5048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:40.920625 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:40.922502 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":122,"threads_started":1}
W20260926 07:25:40.922794 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:40.912739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43641 (local address 127.0.98.62:38995)
0926 07:25:40.912960 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:40.912963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:40.913041 (+ 78us) server_negotiation.cc:407] Connection header received
0926 07:25:40.913185 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:40.913188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:40.913229 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:40.913310 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:40.913775 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:40.914297 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:40.915180 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:40.915391 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:40.915468 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:40.922628 (+ 7160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43641: BlockingRecv error: recv got EOF from 127.0.98.1:43641 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":81,"threads_started":1}
Sep 26 07:25:41 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:41 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:41.103569 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:41.097924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:58561)
0926 07:25:41.098187 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:41.098201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:41.098291 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:41.098604 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:41.098609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:41.098815 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:41.098913 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:41.098918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:41.099664 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:41.099666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:41.100392 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:41.100401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:41.100952 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:41.100956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:41.101041 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:41.101664 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:41.101676 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:41.103447 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
W20260926 07:25:41.103801 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:41.098051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58561 (local address 127.0.98.62:38995)
0926 07:25:41.098236 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:41.098240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:41.098254 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:41.098381 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:41.098383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:41.098425 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:41.098494 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:41.099055 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:41.099570 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:41.100557 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:41.100844 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:41.100959 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:41.103572 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58561: BlockingRecv error: recv got EOF from 127.0.98.2:58561 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 26 07:25:41 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:41 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:41.281558 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:41.276580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:43883)
0926 07:25:41.276832 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:41.276844 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:41.276935 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:41.277151 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:41.277153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:41.277323 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:41.277412 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:41.277417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:41.278115 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:41.278117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:41.278675 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:41.278682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:41.279144 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:41.279147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:41.279208 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:41.279721 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:41.279731 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:41.281429 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
W20260926 07:25:41.281675 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:41.276672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43883 (local address 127.0.98.62:38995)
0926 07:25:41.276798 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:41.276802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:41.276908 (+ 106us) server_negotiation.cc:407] Connection header received
0926 07:25:41.277006 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:41.277008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:41.277050 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:41.277117 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:41.277506 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:41.278027 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:41.278780 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:41.279005 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:41.279106 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:41.281561 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43883: BlockingRecv error: recv got EOF from 127.0.98.3:43883 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 07:25:41 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:41 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:41.929188 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:41.923394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:40913)
0926 07:25:41.923676 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:41.923706 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:41.923845 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:41.924209 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:41.924212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:41.924390 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:41.924498 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:41.924504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:41.925696 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:41.925699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:41.926306 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:41.926315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:41.926742 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:41.926744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:41.926802 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:41.927299 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:41.927310 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:41.929073 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":100,"threads_started":1}
W20260926 07:25:41.929520 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:41.923419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40913 (local address 127.0.98.62:38995)
0926 07:25:41.923762 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:41.923766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:41.923784 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:25:41.923980 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:41.923985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:41.924054 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:41.924152 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:41.924636 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:41.925574 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:41.926424 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:41.926642 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:41.926724 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:41.929327 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40913: BlockingRecv error: recv got EOF from 127.0.98.1:40913 (error 108)
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":102,"threads_started":1}
Sep 26 07:25:42 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:42 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:42.110057 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:42.104420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:33061)
0926 07:25:42.104710 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:42.104734 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:42.104878 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:42.105203 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:42.105206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:42.105420 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:42.105545 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:42.105553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:42.106323 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:42.106327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:42.106954 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:42.106963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:42.107423 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:42.107426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:42.107495 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:42.108057 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:42.108069 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:42.109939 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":106,"threads_started":1}
W20260926 07:25:42.110149 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:42.104451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33061 (local address 127.0.98.62:38995)
0926 07:25:42.104572 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:42.104577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:42.104817 (+ 240us) server_negotiation.cc:407] Connection header received
0926 07:25:42.104997 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:42.105000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:42.105060 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:42.105142 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:42.105672 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:42.106217 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:42.107121 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:42.107327 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:42.107403 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:42.110060 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33061: BlockingRecv error: recv got EOF from 127.0.98.2:33061 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 07:25:42 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:42 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:42.287403 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:42.282338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:60289)
0926 07:25:42.282617 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:42.282630 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:42.282727 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:42.282962 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:42.282964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:42.283156 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:42.283255 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:42.283261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:42.283960 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:42.283962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:42.284503 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:42.284510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:42.284895 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:42.284897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:42.284951 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:42.285511 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:42.285523 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:42.287292 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":111,"threads_started":1}
W20260926 07:25:42.287542 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:42.282437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60289 (local address 127.0.98.62:38995)
0926 07:25:42.282567 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:42.282571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:42.282700 (+ 129us) server_negotiation.cc:407] Connection header received
0926 07:25:42.282805 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:42.282807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:42.282850 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:42.282924 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:42.283353 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:42.283869 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:42.284597 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:42.284802 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:42.284876 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:42.287420 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60289: BlockingRecv error: recv got EOF from 127.0.98.3:60289 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 07:25:42 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:42 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:42.935346 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:42.929988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:47753)
0926 07:25:42.930303 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:42.930317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:42.930401 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:42.930642 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:42.930645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:42.930844 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:42.930943 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:42.930948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:42.931698 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:42.931700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:42.932304 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:42.932314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:42.932711 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:42.932714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:42.932776 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:42.933379 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:42.933391 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:42.935219 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":121,"threads_started":1}
W20260926 07:25:42.935581 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:42.930164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47753 (local address 127.0.98.62:38995)
0926 07:25:42.930399 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:42.930403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:42.930418 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:25:42.930484 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:42.930486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:42.930524 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:42.930603 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:42.931066 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:42.931584 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:42.932410 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:42.932618 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:42.932690 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:42.935418 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47753: BlockingRecv error: recv got EOF from 127.0.98.1:47753 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":93,"threads_started":1}
Sep 26 07:25:43 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:43 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:43.116894 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:43.110988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:44849)
0926 07:25:43.111354 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:43.111372 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:43.111492 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:43.111787 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:43.111790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:43.112003 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:43.112099 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:43.112105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:43.112936 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:43.112943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:43.113582 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:43.113591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:43.114094 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:43.114098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:43.114189 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:43.114700 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:43.114712 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:43.116700 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":133,"threads_started":1}
W20260926 07:25:43.116956 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:43.111173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44849 (local address 127.0.98.62:38995)
0926 07:25:43.111313 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:43.111318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:43.111452 (+ 134us) server_negotiation.cc:407] Connection header received
0926 07:25:43.111603 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:43.111607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:43.111665 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:43.111767 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:43.112227 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:43.112819 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:43.113710 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:43.113969 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:43.114093 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:43.116837 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44849: BlockingRecv error: recv got EOF from 127.0.98.2:44849 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 07:25:43 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:43 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:43.293377 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:43.288324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:33615)
0926 07:25:43.288588 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:43.288601 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:43.288705 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:43.288928 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:43.288930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:43.289103 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:43.289203 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:43.289210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:43.289920 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:43.289922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:43.290516 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:43.290524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:43.290950 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:43.290953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:43.291017 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:43.291516 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:43.291527 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:43.293216 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":96,"threads_started":1}
W20260926 07:25:43.293502 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:43.288377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33615 (local address 127.0.98.62:38995)
0926 07:25:43.288517 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:43.288522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:43.288674 (+ 152us) server_negotiation.cc:407] Connection header received
0926 07:25:43.288785 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:43.288787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:43.288829 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:43.288896 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:43.289300 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:43.289817 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:43.290617 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:43.290820 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:43.290893 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:43.293390 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33615: BlockingRecv error: recv got EOF from 127.0.98.3:33615 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 07:25:43 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:43 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:43.941805 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:43.936250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:42231)
0926 07:25:43.936520 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:43.936533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:43.936635 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:43.936930 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:43.936934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:43.937147 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:43.937244 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:43.937250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:43.937986 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:43.937988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:43.938547 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:43.938554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:43.938990 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:43.938992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:43.939080 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:43.939630 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:43.939642 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:43.941662 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
W20260926 07:25:43.942013 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:43.936311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42231 (local address 127.0.98.62:38995)
0926 07:25:43.936550 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:43.936554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:43.936605 (+ 51us) server_negotiation.cc:407] Connection header received
0926 07:25:43.936726 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:43.936728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:43.936772 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:43.936860 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:43.937356 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:43.937892 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:43.938656 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:43.938887 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:43.938970 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:43.941839 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42231: BlockingRecv error: recv got EOF from 127.0.98.1:42231 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
Sep 26 07:25:44 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:44 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:44.122943 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:44.117626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:35573)
0926 07:25:44.117951 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:44.117963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:44.118053 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:44.118268 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:44.118270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:44.118446 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:44.118541 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:44.118546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:44.119395 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:44.119397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:44.119951 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:44.119959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:44.120361 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:44.120364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:44.120422 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:44.120914 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:44.120926 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:44.122830 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":69,"threads_started":1}
W20260926 07:25:44.123271 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:44.117675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35573 (local address 127.0.98.62:38995)
0926 07:25:44.117870 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:44.117875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:44.118022 (+ 147us) server_negotiation.cc:407] Connection header received
0926 07:25:44.118138 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:44.118146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:44.118188 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:44.118248 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:44.118632 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:44.119275 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:44.120052 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:44.120261 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:44.120355 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:44.123011 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35573: BlockingRecv error: recv got EOF from 127.0.98.2:35573 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 07:25:44 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:44 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:44.300050 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:44.294299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:55723)
0926 07:25:44.294583 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:44.294597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:44.294701 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:44.294949 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:44.294951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:44.295197 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:44.295297 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:44.295303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:44.296014 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:44.296016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:44.296639 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:44.296647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:44.297068 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:44.297071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:44.297162 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:44.297790 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:44.297805 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:44.299940 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
W20260926 07:25:44.300148 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:44.294417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55723 (local address 127.0.98.62:38995)
0926 07:25:44.294555 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:44.294563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:44.294696 (+ 133us) server_negotiation.cc:407] Connection header received
0926 07:25:44.294774 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:44.294776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:44.294833 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:44.294911 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:44.295405 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:44.295921 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:44.296748 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:44.296960 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:44.297039 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:44.300062 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55723: BlockingRecv error: recv got EOF from 127.0.98.3:55723 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 07:25:44 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:44 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:44.948725 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:44.942721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:35971)
0926 07:25:44.943080 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:44.943102 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:44.943201 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:44.943438 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:44.943440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:44.943666 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:44.943770 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:44.943776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:44.944728 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:44.944742 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:44.945389 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:44.945401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:44.945981 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:44.945983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:44.946072 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:44.946618 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:44.946630 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:44.948615 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":91,"threads_started":1}
W20260926 07:25:44.948946 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:44.942793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35971 (local address 127.0.98.62:38995)
0926 07:25:44.943090 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:44.943096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:44.943198 (+ 102us) server_negotiation.cc:407] Connection header received
0926 07:25:44.943276 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:44.943278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:44.943324 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:44.943409 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:44.943889 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:44.944596 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:44.945513 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:44.945849 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:44.945984 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:44.948773 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35971: BlockingRecv error: recv got EOF from 127.0.98.1:35971 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
Sep 26 07:25:45 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:45 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:45.129571 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:45.123847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:52537)
0926 07:25:45.124205 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:45.124219 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:45.124356 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:45.124695 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:45.124697 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:45.124885 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:45.124981 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:45.124986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:45.125711 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:45.125713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:45.126294 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:45.126302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:45.126715 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:45.126717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:45.126777 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:45.127276 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:45.127289 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:45.129402 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":66,"threads_started":1}
W20260926 07:25:45.129631 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:45.123992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52537 (local address 127.0.98.62:38995)
0926 07:25:45.124178 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:45.124181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:45.124303 (+ 122us) server_negotiation.cc:407] Connection header received
0926 07:25:45.124455 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:45.124457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:45.124503 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:45.124672 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:45.125080 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:45.125599 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:45.126394 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:45.126603 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:45.126677 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:45.129525 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52537: BlockingRecv error: recv got EOF from 127.0.98.2:52537 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 07:25:45 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:45 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:45.306555 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:45.301116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:47823)
0926 07:25:45.301494 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:45.301509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:45.301648 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:45.302076 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:45.302078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:45.302303 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:45.302395 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:45.302400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:45.303143 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:45.303145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:45.303702 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:45.303709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:45.304097 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:45.304100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:45.304158 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:45.304638 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:45.304648 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:45.306447 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":129,"threads_started":1}
W20260926 07:25:45.306720 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:45.301165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47823 (local address 127.0.98.62:38995)
0926 07:25:45.301343 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:45.301348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:45.301591 (+ 243us) server_negotiation.cc:407] Connection header received
0926 07:25:45.301773 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:45.301779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:45.301836 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:45.301918 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:45.302496 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:45.303006 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:45.303798 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:45.304007 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:45.304079 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:45.306582 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47823: BlockingRecv error: recv got EOF from 127.0.98.3:47823 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 07:25:45 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:45 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:45.958527 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:45.949591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:43779)
0926 07:25:45.949907 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:45.949920 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:45.950009 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:45.950320 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:45.950322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:45.950566 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:45.950669 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:45.950675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:45.951647 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:45.951650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:45.952225 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:45.952234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:45.952710 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:45.952713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:45.952795 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:45.956509 (+ 3714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:45.956520 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:45.958363 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":145,"threads_started":1}
W20260926 07:25:45.958598 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:45.949768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43779 (local address 127.0.98.62:38995)
0926 07:25:45.949993 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:45.949996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:45.950053 (+ 57us) server_negotiation.cc:407] Connection header received
0926 07:25:45.950146 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:45.950149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:45.950200 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:45.950293 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:45.950790 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:45.951539 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:45.952346 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:45.952582 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:45.952706 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:45.958477 (+ 5771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43779: BlockingRecv error: recv got EOF from 127.0.98.1:43779 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":55,"threads_started":1}
Sep 26 07:25:46 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:46 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:46.135996 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:46.130315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:49871)
0926 07:25:46.130586 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:46.130600 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:46.130695 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:46.130935 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:46.130937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:46.131145 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:46.131242 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:46.131248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:46.132180 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:46.132182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:46.132724 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:46.132731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:46.133150 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:46.133152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:46.133212 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:46.133703 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:46.133714 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:46.135846 (+ 2132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":67,"threads_started":1}
W20260926 07:25:46.136317 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:46.130444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49871 (local address 127.0.98.62:38995)
0926 07:25:46.130629 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:46.130634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:46.130650 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:25:46.130782 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:46.130784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:46.130827 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:46.130897 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:46.131395 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:46.132071 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:46.132820 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:46.133042 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:46.133119 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:46.136029 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49871: BlockingRecv error: recv got EOF from 127.0.98.2:49871 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 07:25:46 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:46 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:46.312501 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:46.307353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:60551)
0926 07:25:46.307603 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:46.307615 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:46.307709 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:46.307935 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:46.307937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:46.308136 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:46.308229 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:46.308234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:46.308911 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:46.308914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:46.309489 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:46.309496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:46.309914 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:46.309921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:46.310009 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:46.310466 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:46.310477 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:46.312362 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":67,"threads_started":1}
W20260926 07:25:46.312594 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:46.307429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60551 (local address 127.0.98.62:38995)
0926 07:25:46.307598 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:46.307602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:46.307677 (+ 75us) server_negotiation.cc:407] Connection header received
0926 07:25:46.307790 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:46.307792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:46.307838 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:46.307903 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:46.308318 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:46.308828 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:46.309598 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:46.309822 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:46.309902 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:46.312475 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60551: BlockingRecv error: recv got EOF from 127.0.98.3:60551 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 07:25:46 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:46 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:46.964960 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:46.959321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:33891)
0926 07:25:46.959570 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:46.959584 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:46.959667 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:46.959902 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:46.959904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:46.960078 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:46.960172 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:46.960178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:46.960860 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:46.960862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:46.961430 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:46.961438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:46.961824 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:46.961826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:46.961883 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:46.962385 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:46.962397 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:46.964798 (+ 2401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
W20260926 07:25:46.965024 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:46.959393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33891 (local address 127.0.98.62:38995)
0926 07:25:46.959614 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:46.959618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:46.959631 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:25:46.959749 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:46.959751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:46.959792 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:46.959869 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:46.960265 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:46.960775 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:46.961524 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:46.961735 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:46.961806 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:46.964925 (+ 3119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33891: BlockingRecv error: recv got EOF from 127.0.98.1:33891 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
Sep 26 07:25:47 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:47 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:47.142194 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:47.136806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:52083)
0926 07:25:47.137138 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:47.137154 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:47.137318 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:47.137586 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:47.137589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:47.137762 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:47.137856 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:47.137861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:47.138626 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:47.138628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:47.139298 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:47.139307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:47.139733 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:47.139736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:47.139793 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:47.140313 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:47.140324 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:47.142072 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":102,"threads_started":1}
W20260926 07:25:47.142319 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:47.136844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52083 (local address 127.0.98.62:38995)
0926 07:25:47.137160 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:47.137164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:47.137270 (+ 106us) server_negotiation.cc:407] Connection header received
0926 07:25:47.137398 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:47.137400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:47.137442 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:47.137522 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:47.137983 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:47.138532 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:47.139402 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:47.139635 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:47.139717 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:47.142204 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52083: BlockingRecv error: recv got EOF from 127.0.98.2:52083 (error 108)
Metrics: {"server-negotiator.queue_time_us":229}
Sep 26 07:25:47 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:47 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:47.318961 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:47.313321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:57071)
0926 07:25:47.313608 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:47.313622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:47.313727 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:47.314012 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:47.314014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:47.314205 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:47.314304 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:47.314310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:47.315120 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:47.315127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:47.315763 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:47.315773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:47.316210 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:47.316213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:47.316298 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:47.316778 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:47.316791 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:47.318824 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":109,"threads_started":1}
W20260926 07:25:47.319152 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:47.313419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57071 (local address 127.0.98.62:38995)
0926 07:25:47.313550 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:47.313553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:47.313693 (+ 140us) server_negotiation.cc:407] Connection header received
0926 07:25:47.313809 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:47.313811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:47.313855 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:47.313922 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:47.314405 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:47.314911 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:47.315886 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:47.316103 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:47.316177 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:47.318947 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57071: BlockingRecv error: recv got EOF from 127.0.98.3:57071 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 07:25:47 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:47 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:47.971634 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:47.965868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:49695)
0926 07:25:47.966162 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:47.966177 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:47.966263 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:47.966484 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:47.966486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:47.966656 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:47.966750 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:47.966755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:47.967630 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:47.967632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:47.968190 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:47.968198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:47.968804 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:47.968808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:47.968883 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:47.969404 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:47.969424 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:47.971464 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
W20260926 07:25:47.971709 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:47.966004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49695 (local address 127.0.98.62:38995)
0926 07:25:47.966245 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:47.966249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:47.966264 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:25:47.966321 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:47.966324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:47.966364 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:47.966452 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:47.966921 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:47.967528 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:47.968317 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:47.968663 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:47.968775 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:47.971588 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49695: BlockingRecv error: recv got EOF from 127.0.98.1:49695 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":61,"threads_started":1}
Sep 26 07:25:48 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:48 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:48.148900 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:48.142986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:56823)
0926 07:25:48.143321 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:48.143361 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:48.143487 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:48.143756 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:48.143759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:48.144009 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:48.144100 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:48.144105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:48.144861 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:48.144866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:48.145428 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:48.145436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:48.145919 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:48.145922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:48.146009 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:48.146631 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:48.146669 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:48.148774 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":96,"threads_started":1}
W20260926 07:25:48.149137 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:48.143108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56823 (local address 127.0.98.62:38995)
0926 07:25:48.143277 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:48.143281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:48.143452 (+ 171us) server_negotiation.cc:407] Connection header received
0926 07:25:48.143595 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:48.143599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:48.143651 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:48.143738 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:48.144221 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:48.144746 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:48.145569 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:48.145798 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:48.145918 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:48.148916 (+ 2998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56823: BlockingRecv error: recv got EOF from 127.0.98.2:56823 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 07:25:48 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:48 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:48.325493 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:48.319725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:41849)
0926 07:25:48.320039 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:48.320057 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:48.320207 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:48.320462 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:48.320465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:48.320648 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:48.320747 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:48.320752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:48.321492 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:48.321495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:48.322136 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:48.322149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:48.322588 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:48.322591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:48.322649 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:48.323232 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:48.323248 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:48.325314 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":232,"threads_started":1}
W20260926 07:25:48.325563 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:48.319859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41849 (local address 127.0.98.62:38995)
0926 07:25:48.319992 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:48.319996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:48.320148 (+ 152us) server_negotiation.cc:407] Connection header received
0926 07:25:48.320287 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:48.320289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:48.320336 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:48.320402 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:48.320869 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:48.321390 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:48.322288 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:48.322501 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:48.322575 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:48.325442 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41849: BlockingRecv error: recv got EOF from 127.0.98.3:41849 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 07:25:48 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:48 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:48.977970 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:48.972695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:57189)
0926 07:25:48.972962 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:48.972976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:48.973073 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:48.973304 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:48.973306 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:48.973480 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:48.973576 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:48.973581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:48.974445 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:48.974448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:48.975006 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:48.975014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:48.975487 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:48.975490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:48.975556 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:48.976031 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:48.976044 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:48.977858 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":66,"threads_started":1}
W20260926 07:25:48.978112 9622 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:48.972616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57189 (local address 127.0.98.62:38995)
0926 07:25:48.972968 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:48.972972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:48.973040 (+ 68us) server_negotiation.cc:407] Connection header received
0926 07:25:48.973146 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:48.973149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:48.973190 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:48.973264 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:48.973666 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:48.974325 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:48.975169 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:48.975384 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:48.975464 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:48.977985 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57189: BlockingRecv error: recv got EOF from 127.0.98.1:57189 (error 108)
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":97,"threads_started":1}
Sep 26 07:25:49 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:49 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:49.155331 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:49.149744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:39279)
0926 07:25:49.150062 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:49.150076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:49.150195 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:49.150508 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:49.150511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:49.150791 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:49.150881 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:49.150887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:49.151703 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:49.151708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:49.152300 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:49.152308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:49.152771 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:49.152774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:49.152861 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:49.153325 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:49.153336 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:49.155179 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":85,"threads_started":1}
W20260926 07:25:49.155535 9622 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:49.149776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39279 (local address 127.0.98.62:38995)
0926 07:25:49.149953 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:49.149958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:49.150139 (+ 181us) server_negotiation.cc:407] Connection header received
0926 07:25:49.150321 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:49.150324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:49.150399 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:49.150474 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:49.151013 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:49.151589 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:49.152421 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:49.152658 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:49.152769 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:49.155319 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39279: BlockingRecv error: recv got EOF from 127.0.98.2:39279 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 07:25:49 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:49 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:49.332345 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:49.326355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:47383)
0926 07:25:49.326676 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:49.326697 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:49.326836 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:49.327152 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:49.327154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:49.327367 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:49.327462 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:49.327467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:49.328270 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:49.328273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:49.328979 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:49.328989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:49.329490 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:49.329493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:49.329583 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:49.330211 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:49.330225 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:49.332206 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
W20260926 07:25:49.332433 9622 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:49.326484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47383 (local address 127.0.98.62:38995)
0926 07:25:49.326646 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:49.326651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:49.326776 (+ 125us) server_negotiation.cc:407] Connection header received
0926 07:25:49.326947 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:49.326952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:49.327001 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:49.327135 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:49.327582 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:49.328141 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:49.329116 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:49.329387 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:49.329471 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:49.332331 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47383: BlockingRecv error: recv got EOF from 127.0.98.3:47383 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 07:25:49 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:49 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:49.984189 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:49.978794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:33639)
0926 07:25:49.979109 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:49.979124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:49.979261 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:49.979623 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:49.979626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:49.979823 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:49.979934 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:49.979941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:49.980700 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:49.980704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:49.981266 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:49.981275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:49.981689 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:49.981693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:49.981760 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:49.982245 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:49.982256 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:49.984070 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":79,"threads_started":1}
W20260926 07:25:49.984390 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:49.978789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33639 (local address 127.0.98.62:38995)
0926 07:25:49.979082 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:49.979087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:49.979204 (+ 117us) server_negotiation.cc:407] Connection header received
0926 07:25:49.979365 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:49.979369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:49.979431 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:49.979539 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:49.980040 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:49.980594 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:49.981366 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:49.981576 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:49.981650 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:49.984231 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33639: BlockingRecv error: recv got EOF from 127.0.98.1:33639 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
Sep 26 07:25:50 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:50 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:50.162267 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:50.156063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:42865)
0926 07:25:50.156371 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:50.156384 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:50.156478 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:50.156755 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:50.156757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:50.156923 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:50.157014 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:50.157019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:50.158247 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:50.158249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:50.158799 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:50.158807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:50.159611 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:50.159615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:50.159706 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:50.160366 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:50.160377 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:50.162140 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":71,"threads_started":1}
W20260926 07:25:50.162503 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:50.156184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42865 (local address 127.0.98.62:38995)
0926 07:25:50.156362 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:50.156370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:50.156461 (+ 91us) server_negotiation.cc:407] Connection header received
0926 07:25:50.156575 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:50.156580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:50.156644 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:50.156735 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:50.157137 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:50.158127 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:50.158923 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:50.159398 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:50.159520 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:50.162338 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42865: BlockingRecv error: recv got EOF from 127.0.98.2:42865 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 07:25:50 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:50 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:50.338537 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:50.333072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:35455)
0926 07:25:50.333348 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:50.333364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:50.333506 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:50.333776 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:50.333779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:50.333989 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:50.334077 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:50.334083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:50.334823 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:50.334829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:50.335421 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:50.335428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:50.335876 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:50.335880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:50.335997 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:50.336556 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:50.336566 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:50.338425 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":73,"threads_started":1}
W20260926 07:25:50.338703 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:50.333137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35455 (local address 127.0.98.62:38995)
0926 07:25:50.333276 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:50.333280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:50.333440 (+ 160us) server_negotiation.cc:407] Connection header received
0926 07:25:50.333611 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:50.333616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:50.333670 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:50.333754 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:50.334191 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:50.334707 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:50.335538 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:50.335759 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:50.335875 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:50.338553 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35455: BlockingRecv error: recv got EOF from 127.0.98.3:35455 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 07:25:50 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:50 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:50.993868 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:50.985013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:53849)
0926 07:25:50.985333 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:50.985356 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:50.985451 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:50.985687 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:50.985689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:50.985863 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:50.985957 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:50.985962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:50.986673 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:50.986676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:50.987253 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:50.987262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:50.987733 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:50.987735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:50.987793 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:50.991867 (+ 4074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:50.991880 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:50.993746 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":80,"threads_started":1}
W20260926 07:25:50.994063 9631 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:50.985070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53849 (local address 127.0.98.62:38995)
0926 07:25:50.985352 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:50.985357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:50.985419 (+ 62us) server_negotiation.cc:407] Connection header received
0926 07:25:50.985531 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:50.985533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:50.985575 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:50.985654 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:50.986055 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:50.986583 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:50.987401 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:50.987628 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:50.987720 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:50.993896 (+ 6176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53849: BlockingRecv error: recv got EOF from 127.0.98.1:53849 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":60,"threads_started":1}
Sep 26 07:25:51 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:51 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:51.168350 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:51.163152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:57383)
0926 07:25:51.163474 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:51.163487 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:51.163589 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:51.163831 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:51.163833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:51.164004 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:51.164096 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:51.164100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:51.164836 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:51.164838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:51.165400 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:51.165408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:51.165862 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:51.165864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:51.165922 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:51.166423 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:51.166434 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:51.168239 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":128,"threads_started":1}
W20260926 07:25:51.168501 9631 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:51.163266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57383 (local address 127.0.98.62:38995)
0926 07:25:51.163466 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:51.163469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:51.163563 (+ 94us) server_negotiation.cc:407] Connection header received
0926 07:25:51.163682 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:51.163685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:51.163727 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:51.163798 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:51.164189 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:51.164747 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:51.165513 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:51.165763 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:51.165844 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:51.168353 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57383: BlockingRecv error: recv got EOF from 127.0.98.2:57383 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 07:25:51 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:51 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:51.345026 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:51.339316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:40759)
0926 07:25:51.339651 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:51.339665 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:51.339765 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:51.340009 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:51.340010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:51.340195 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:51.340288 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:51.340293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:51.341033 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:51.341035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:51.341582 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:51.341589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:51.341994 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:51.341996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:51.342055 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:51.342654 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:51.342666 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:51.344857 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":151,"threads_started":1}
W20260926 07:25:51.345135 9631 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:51.339585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40759 (local address 127.0.98.62:38995)
0926 07:25:51.339725 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:51.339728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:51.339742 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:25:51.339850 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:51.339852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:51.339908 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:51.339973 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:51.340393 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:51.340935 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:51.341687 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:51.341899 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:51.341977 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:51.344971 (+ 2994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40759: BlockingRecv error: recv got EOF from 127.0.98.3:40759 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 07:25:51 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:51 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:52.000370 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:51.994670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:58173)
0926 07:25:51.994970 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:51.994986 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:51.995145 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:51.995443 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:51.995449 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:51.995703 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:51.995808 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:51.995814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:51.996565 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:51.996568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:51.997218 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:51.997227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:51.997640 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:51.997642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:51.997702 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:51.998368 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:51.998380 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:52.000248 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":120,"threads_started":1}
W20260926 07:25:52.000488 9635 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:51.994728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58173 (local address 127.0.98.62:38995)
0926 07:25:51.995025 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:51.995055 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:51.995075 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:25:51.995253 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:51.995256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:51.995307 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:51.995402 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:51.995915 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:51.996443 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:51.997323 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:51.997537 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:51.997615 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:52.000368 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58173: BlockingRecv error: recv got EOF from 127.0.98.1:58173 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
Sep 26 07:25:52 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:52 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:52.174373 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:52.169156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:35609)
0926 07:25:52.169509 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:52.169522 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:52.169618 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:52.169875 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:52.169877 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:52.170042 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:52.170148 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:52.170154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:52.170925 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:52.170927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:52.171509 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:52.171517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:52.171940 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:52.171942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:52.172002 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:52.172543 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:52.172554 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:52.174261 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":109,"threads_started":1}
W20260926 07:25:52.174557 9635 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:52.169297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35609 (local address 127.0.98.62:38995)
0926 07:25:52.169493 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:52.169497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:52.169600 (+ 103us) server_negotiation.cc:407] Connection header received
0926 07:25:52.169717 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:52.169720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:52.169765 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:52.169841 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:52.170274 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:52.170831 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:52.171628 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:52.171843 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:52.171927 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:52.174415 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35609: BlockingRecv error: recv got EOF from 127.0.98.2:35609 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 07:25:52 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:52 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:52.351713 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:52.345947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.3:56489)
0926 07:25:52.346259 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:52.346273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:52.346403 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:52.346711 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:52.346714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:52.346917 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:52.347007 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:52.347021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:52.347792 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:52.347799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:52.348369 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:52.348376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:52.348826 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:52.348829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:52.348913 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:52.349408 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:52.349419 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:52.351588 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":104,"threads_started":1}
W20260926 07:25:52.351979 9635 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:52.346005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56489 (local address 127.0.98.62:38995)
0926 07:25:52.346188 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:52.346192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:52.346349 (+ 157us) server_negotiation.cc:407] Connection header received
0926 07:25:52.346510 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:52.346515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:52.346593 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:52.346687 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:52.347162 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:52.347686 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:52.348485 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:52.348714 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:52.348822 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:52.351770 (+ 2948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56489: BlockingRecv error: recv got EOF from 127.0.98.3:56489 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 07:25:53 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:53 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:53.006700 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:53.001293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.1:47581)
0926 07:25:53.001552 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:53.001571 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:53.001673 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:53.001954 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:53.001957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:53.002149 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:53.002243 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:53.002249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.002987 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.002990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:53.003638 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:53.003647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.004103 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.004106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:53.004169 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:53.004681 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:53.004692 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:53.006579 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":63,"threads_started":1}
W20260926 07:25:53.006940 9639 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:53.001475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47581 (local address 127.0.98.62:38995)
0926 07:25:53.001718 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:53.001722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:53.001734 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:25:53.001766 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:53.001768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:53.001805 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:53.001881 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:53.002350 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.002871 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.003756 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.003999 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.004083 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:53.006789 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47581: BlockingRecv error: recv got EOF from 127.0.98.1:47581 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
Sep 26 07:25:53 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:25:53 dist-test-slave-g04g krb5kdc[8581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:25:53.180765 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:53.175324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38995 (local address 127.0.98.2:36131)
0926 07:25:53.175608 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:53.175621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:53.175712 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:53.175924 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:53.175926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:53.176094 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:53.176190 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:53.176195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.176882 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.176884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:53.177436 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:53.177443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.177826 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.177829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:53.177887 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:53.178343 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:53.178354 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:53.180646 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38995: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":101,"threads_started":1}
W20260926 07:25:53.180924 9639 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:53.175388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36131 (local address 127.0.98.62:38995)
0926 07:25:53.175532 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:53.175536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:53.175686 (+ 150us) server_negotiation.cc:407] Connection header received
0926 07:25:53.175776 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:53.175778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:53.175818 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:53.175887 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:53.176284 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.176792 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.177532 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.177735 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.177807 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:53.180774 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36131: BlockingRecv error: recv got EOF from 127.0.98.2:36131 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:25:53.249097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 9164
I20260926 07:25:53.254515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 9316
I20260926 07:25:53.259660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 9448
I20260926 07:25:53.264596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 9094
2026-09-26T07:25:53Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17939 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T07:25:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:53Z Disabled control of system clock
I20260926 07:25:53.294777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41721
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 07:25:53.403738 9644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:53.404006 9644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:53.404038 9644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:53.408216 9644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:53.408277 9644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:53.408291 9644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:53.408308 9644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:53.408324 9644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:53.413071 9644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:53.414085 9644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:53.415496 9644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:53.423724 9649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:53.423771 9650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:53.423745 9652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:53.424057 9644 server_base.cc:1034] running on GCE node
I20260926 07:25:53.424621 9644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:53.425652 9644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:53.426821 9644 hybrid_clock.cc:648] HybridClock initialized: now 1790407553426791 us; error 48 us; skew 500 ppm
I20260926 07:25:53.428670 9644 webserver.cc:460] Webserver started at http://127.0.98.62:35683/ using document root <none> and password file <none>
I20260926 07:25:53.429236 9644 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:53.429296 9644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:53.429538 9644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:53.431592 9644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "02778a5e157f499f80d063b00a2a5126"
format_stamp: "Formatted at 2026-09-26 07:25:53 on dist-test-slave-g04g"
server_key: "78fb6c81127662b92189f89127663338"
server_key_iv: "d8ec7e57b92533ab90cb4967bb2b69ed"
server_key_version: "encryptionkey@0"
I20260926 07:25:53.432155 9644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "02778a5e157f499f80d063b00a2a5126"
format_stamp: "Formatted at 2026-09-26 07:25:53 on dist-test-slave-g04g"
server_key: "78fb6c81127662b92189f89127663338"
server_key_iv: "d8ec7e57b92533ab90cb4967bb2b69ed"
server_key_version: "encryptionkey@0"
I20260926 07:25:53.435781 9644 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:53.438225 9658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:53.439659 9644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:53.439800 9644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "02778a5e157f499f80d063b00a2a5126"
format_stamp: "Formatted at 2026-09-26 07:25:53 on dist-test-slave-g04g"
server_key: "78fb6c81127662b92189f89127663338"
server_key_iv: "d8ec7e57b92533ab90cb4967bb2b69ed"
server_key_version: "encryptionkey@0"
I20260926 07:25:53.439927 9644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:53.461086 9644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:53.461757 9644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:53.461956 9644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:53.468827 9644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41721
I20260926 07:25:53.468834 9710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41721 every 8 connection(s)
I20260926 07:25:53.469750 9644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:53.470319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 9644
I20260926 07:25:53.470485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:53.470780 392 external_mini_cluster.cc:1428] Setting key 52d146ab385c48930ba3d2bb0d4c1912
I20260926 07:25:53.473212 9711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:53.479370 9711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126: Bootstrap starting.
I20260926 07:25:53.480545 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:53.472371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.0.98.62:41721)
0926 07:25:53.473074 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:53.473091 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:53.473127 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:25:53.473867 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:53.473885 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:53.474298 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:53.474602 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:53.474950 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.476054 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.477257 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.477745 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.477885 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:53.478715 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:53.478720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:53.478731 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:53.479201 (+ 470us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:53.479256 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:53.479277 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:53.479353 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:53.479550 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 07:25:53.479840 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":167,"threads_started":1}
I20260926 07:25:53.481599 9711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:53.482327 9711 log.cc:824] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:53.484189 9711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126: No bootstrap required, opened a new log
I20260926 07:25:53.487774 9711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02778a5e157f499f80d063b00a2a5126" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41721 } }
I20260926 07:25:53.488010 9711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:53.488101 9711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02778a5e157f499f80d063b00a2a5126, State: Initialized, Role: FOLLOWER
I20260926 07:25:53.488513 9711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02778a5e157f499f80d063b00a2a5126" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41721 } }
I20260926 07:25:53.488672 9711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:53.488754 9711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:53.488867 9711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:53.489717 9711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02778a5e157f499f80d063b00a2a5126" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41721 } }
I20260926 07:25:53.490042 9711 leader_election.cc:302] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02778a5e157f499f80d063b00a2a5126; no voters:
I20260926 07:25:53.490329 9711 leader_election.cc:288] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:53.490566 9716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:53.490846 9716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [term 1 LEADER]: Becoming Leader. State: Replica: 02778a5e157f499f80d063b00a2a5126, State: Running, Role: LEADER
I20260926 07:25:53.491480 9716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02778a5e157f499f80d063b00a2a5126" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41721 } }
I20260926 07:25:53.491673 9711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:53.493331 9717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02778a5e157f499f80d063b00a2a5126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02778a5e157f499f80d063b00a2a5126" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41721 } } }
I20260926 07:25:53.493464 9717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:53.493723 9718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02778a5e157f499f80d063b00a2a5126. Latest consensus state: current_term: 1 leader_uuid: "02778a5e157f499f80d063b00a2a5126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02778a5e157f499f80d063b00a2a5126" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41721 } } }
I20260926 07:25:53.493811 9718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:53.493832 9730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:53.497140 9730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:53.502501 9730 catalog_manager.cc:1347] Generated new cluster ID: ad05992eeb294f4b9e1e4c61ce4291cd
I20260926 07:25:53.502571 9730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:53.520973 9730 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:53.521922 9730 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:53.530563 9730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02778a5e157f499f80d063b00a2a5126: Generated new TSK 0
I20260926 07:25:53.531355 9730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:25:53.539009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41721
--builtin_ntp_servers=127.0.98.20:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 07:25:53.649407 9735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:53.649657 9735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:53.649693 9735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:53.653638 9735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:53.653688 9735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:53.653739 9735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:53.658449 9735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:53.659528 9735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:53.660863 9735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:53.670409 9740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:53.670462 9741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:53.670564 9735 server_base.cc:1034] running on GCE node
W20260926 07:25:53.670465 9743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:53.671170 9735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:53.671816 9735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:53.673009 9735 hybrid_clock.cc:648] HybridClock initialized: now 1790407553672994 us; error 27 us; skew 500 ppm
I20260926 07:25:53.674803 9735 webserver.cc:460] Webserver started at http://127.0.98.1:38169/ using document root <none> and password file <none>
I20260926 07:25:53.675424 9735 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:53.675518 9735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:53.675741 9735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:53.677721 9735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "3d2e262d50df42e98e8e176b49647090"
format_stamp: "Formatted at 2026-09-26 07:25:53 on dist-test-slave-g04g"
server_key: "b73ec2627428c042f1dc87343ddeb1e9"
server_key_iv: "22c5f86f46597fb98ce13aed8681d246"
server_key_version: "encryptionkey@0"
I20260926 07:25:53.678287 9735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3d2e262d50df42e98e8e176b49647090"
format_stamp: "Formatted at 2026-09-26 07:25:53 on dist-test-slave-g04g"
server_key: "b73ec2627428c042f1dc87343ddeb1e9"
server_key_iv: "22c5f86f46597fb98ce13aed8681d246"
server_key_version: "encryptionkey@0"
I20260926 07:25:53.681931 9735 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:25:53.684612 9749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:53.685822 9735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:53.685956 9735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "3d2e262d50df42e98e8e176b49647090"
format_stamp: "Formatted at 2026-09-26 07:25:53 on dist-test-slave-g04g"
server_key: "b73ec2627428c042f1dc87343ddeb1e9"
server_key_iv: "22c5f86f46597fb98ce13aed8681d246"
server_key_version: "encryptionkey@0"
I20260926 07:25:53.686069 9735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:53.702513 9735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:53.703266 9735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:53.703464 9735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:53.704038 9735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:53.704999 9735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:53.705089 9735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:53.705159 9735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:53.705206 9735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:53.713495 9735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41327
I20260926 07:25:53.713564 9862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41327 every 8 connection(s)
I20260926 07:25:53.714377 9735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:53.715251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 9735
I20260926 07:25:53.715374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:25:53.715653 392 external_mini_cluster.cc:1428] Setting key 9d14e8485e02ea68dbf6ad1e17f49bc3
I20260926 07:25:53.717691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41721
--builtin_ntp_servers=127.0.98.20:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 07:25:53.725098 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:53.716298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36341 (local address 127.0.98.62:41721)
0926 07:25:53.716444 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:53.716448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:53.717844 (+ 1396us) server_negotiation.cc:407] Connection header received
0926 07:25:53.719732 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:53.719735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:53.719789 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:53.719867 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:53.720704 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.721591 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.722779 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.723242 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.723376 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:53.724460 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:53.724465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:53.724469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:53.724488 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:53.724512 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:53.724525 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:53.724612 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:53.724772 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 07:25:53.724863 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:25:53.725833 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:53.716759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41721 (local address 127.0.98.1:36341)
0926 07:25:53.717653 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:53.717708 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:53.719532 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:53.720027 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:53.720033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:53.720112 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:53.720568 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:53.720580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.721731 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.721734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:53.722654 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:53.722663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.723428 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.723431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:53.723563 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:53.724196 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:53.724198 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:53.724235 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:53.724305 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:53.724638 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:53.724644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:53.724647 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:25:53.724738 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:25:53.724941 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":132,"threads_started":1}
I20260926 07:25:53.727003 9863 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41721
I20260926 07:25:53.727250 9863 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:53.727725 9863 heartbeater.cc:511] Master 127.0.98.62:41721 requested a full tablet report, sending...
I20260926 07:25:53.729095 9675 ts_manager.cc:194] Registered new tserver with Master: 3d2e262d50df42e98e8e176b49647090 (127.0.98.1:41327)
I20260926 07:25:53.730188 9675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36341
W20260926 07:25:53.827858 9866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:53.828135 9866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:53.828200 9866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:53.832139 9866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:53.832218 9866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:53.832304 9866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:53.837132 9866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:53.838195 9866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:53.839619 9866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:25:53.849263 9866 server_base.cc:1034] running on GCE node
W20260926 07:25:53.849294 9874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:53.849294 9871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:53.849267 9872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:53.849951 9866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:53.850595 9866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:53.851771 9866 hybrid_clock.cc:648] HybridClock initialized: now 1790407553851757 us; error 36 us; skew 500 ppm
I20260926 07:25:53.853777 9866 webserver.cc:460] Webserver started at http://127.0.98.2:37899/ using document root <none> and password file <none>
I20260926 07:25:53.854398 9866 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:53.854462 9866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:53.854683 9866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:53.856986 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "2efa3001e44f4cce89d3ef4b3d9175a3"
format_stamp: "Formatted at 2026-09-26 07:25:53 on dist-test-slave-g04g"
server_key: "e0024b7187221da7c0dd34db94511b4d"
server_key_iv: "9c529dc472f97fcbec86b70809589d1e"
server_key_version: "encryptionkey@0"
I20260926 07:25:53.857609 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2efa3001e44f4cce89d3ef4b3d9175a3"
format_stamp: "Formatted at 2026-09-26 07:25:53 on dist-test-slave-g04g"
server_key: "e0024b7187221da7c0dd34db94511b4d"
server_key_iv: "9c529dc472f97fcbec86b70809589d1e"
server_key_version: "encryptionkey@0"
I20260926 07:25:53.861601 9866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 07:25:53.864519 9880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:53.865971 9866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:25:53.866111 9866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "2efa3001e44f4cce89d3ef4b3d9175a3"
format_stamp: "Formatted at 2026-09-26 07:25:53 on dist-test-slave-g04g"
server_key: "e0024b7187221da7c0dd34db94511b4d"
server_key_iv: "9c529dc472f97fcbec86b70809589d1e"
server_key_version: "encryptionkey@0"
I20260926 07:25:53.866235 9866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:53.879714 9866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:53.880396 9866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:53.880609 9866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:53.881219 9866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:53.882200 9866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:53.882277 9866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:53.882369 9866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:53.882429 9866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:53.892872 9866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42369
I20260926 07:25:53.892896 9993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42369 every 8 connection(s)
I20260926 07:25:53.893865 9866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:25:53.894837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 9866
I20260926 07:25:53.894963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:25:53.895284 392 external_mini_cluster.cc:1428] Setting key ca28615bad08378deaf71ef1be7b3167
I20260926 07:25:53.897603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41721
--builtin_ntp_servers=127.0.98.20:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 07:25:53.905318 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:53.897066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34569 (local address 127.0.98.62:41721)
0926 07:25:53.897197 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:53.897202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:53.898604 (+ 1402us) server_negotiation.cc:407] Connection header received
0926 07:25:53.900175 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:53.900179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:53.900235 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:53.900339 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:53.901364 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.901934 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.903226 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.903473 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.903594 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:53.904787 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:53.904791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:53.904794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:53.904811 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:53.904829 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:53.904837 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:53.904919 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:53.905117 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 07:25:53.905173 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:25:53.906139 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:53.897619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41721 (local address 127.0.98.2:34569)
0926 07:25:53.898452 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:53.898506 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:53.899958 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:53.900528 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:53.900535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:53.900622 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:53.901208 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:53.901223 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.902072 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.902083 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:53.903094 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:53.903110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:53.903629 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:53.903634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:53.903822 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:53.904510 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:53.904512 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:53.904550 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:53.904609 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:53.904922 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:53.904933 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:53.904939 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:25:53.905117 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 07:25:53.905379 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":162,"threads_started":1}
I20260926 07:25:53.907392 9994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41721
I20260926 07:25:53.907620 9994 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:53.908172 9994 heartbeater.cc:511] Master 127.0.98.62:41721 requested a full tablet report, sending...
I20260926 07:25:53.909276 9675 ts_manager.cc:194] Registered new tserver with Master: 2efa3001e44f4cce89d3ef4b3d9175a3 (127.0.98.2:42369)
I20260926 07:25:53.910000 9675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34569
W20260926 07:25:54.009262 9997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:54.009557 9997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:54.009596 9997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:54.014215 9997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:54.014268 9997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:54.014323 9997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:54.018971 9997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:54.020027 9997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:54.021433 9997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:54.031086 10002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:54.031096 10003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:54.031162 10005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:54.031471 9997 server_base.cc:1034] running on GCE node
I20260926 07:25:54.031788 9997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:54.032316 9997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:54.033511 9997 hybrid_clock.cc:648] HybridClock initialized: now 1790407554033482 us; error 45 us; skew 500 ppm
I20260926 07:25:54.035401 9997 webserver.cc:460] Webserver started at http://127.0.98.3:46873/ using document root <none> and password file <none>
I20260926 07:25:54.035946 9997 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:54.036058 9997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:54.036335 9997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:54.038625 9997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "2a804b77b3f44eaba9e883a6d36ab54b"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "31c0f7821f2f74e510c4adf3fde55028"
server_key_iv: "e753f617b171b0b94a3961e3f03f11d4"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.039225 9997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2a804b77b3f44eaba9e883a6d36ab54b"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "31c0f7821f2f74e510c4adf3fde55028"
server_key_iv: "e753f617b171b0b94a3961e3f03f11d4"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.042686 9997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:54.045208 10011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:54.046473 9997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:25:54.046586 9997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "2a804b77b3f44eaba9e883a6d36ab54b"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "31c0f7821f2f74e510c4adf3fde55028"
server_key_iv: "e753f617b171b0b94a3961e3f03f11d4"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.046687 9997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:54.069845 9997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:54.070479 9997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:54.070675 9997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:54.071264 9997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:54.072149 9997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:54.072193 9997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:54.072263 9997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:54.072299 9997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:54.080457 9997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40633
I20260926 07:25:54.080489 10124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40633 every 8 connection(s)
I20260926 07:25:54.081310 9997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:54.083621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 9997
I20260926 07:25:54.083732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:54.084034 392 external_mini_cluster.cc:1428] Setting key 1beadda835055ecf3aee87d9d7cf7a02
I20260926 07:25:54.090906 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:54.083079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33053 (local address 127.0.98.62:41721)
0926 07:25:54.083234 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:54.083239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:54.084445 (+ 1206us) server_negotiation.cc:407] Connection header received
0926 07:25:54.086223 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:54.086226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:54.086289 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:54.086388 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:54.087291 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:54.087811 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:54.089087 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:54.089306 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:54.089412 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:54.090448 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:54.090451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:54.090456 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:54.090473 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:54.090491 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:54.090498 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:54.090575 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:54.090715 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 07:25:54.090769 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:25:54.091715 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:54.083544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41721 (local address 127.0.98.3:33053)
0926 07:25:54.084263 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:54.084327 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:54.086018 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:54.086563 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:54.086570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:54.086647 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:54.087165 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:54.087179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:54.087935 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:54.087938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:54.088974 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:54.088982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:54.089443 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:54.089447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:54.089654 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:54.090207 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:54.090209 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:54.090246 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:54.090303 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:54.090576 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:54.090587 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:54.090596 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:25:54.090715 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 07:25:54.090976 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":139,"threads_started":1}
I20260926 07:25:54.092697 10125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41721
I20260926 07:25:54.092922 10125 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:54.093402 10125 heartbeater.cc:511] Master 127.0.98.62:41721 requested a full tablet report, sending...
I20260926 07:25:54.094303 9675 ts_manager.cc:194] Registered new tserver with Master: 2a804b77b3f44eaba9e883a6d36ab54b (127.0.98.3:40633)
I20260926 07:25:54.095140 9675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33053
I20260926 07:25:54.097813 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:25:54.100443 9714 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:54.099653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56630 (local address 127.0.98.62:41721)
0926 07:25:54.099782 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:54.099787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:54.099918 (+ 131us) server_negotiation.cc:407] Connection header received
0926 07:25:54.100005 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:54.100007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:54.100041 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:54.100097 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:54.100309 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56630: BlockingRecv error: recv got EOF from 127.0.0.1:56630 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:25:54.100536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 9735
I20260926 07:25:54.106670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 9866
I20260926 07:25:54.111980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 9997
I20260926 07:25:54.116865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 9644
2026-09-26T07:25:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (851 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T07:25:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:54Z Disabled control of system clock
I20260926 07:25:54.145843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43105
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:25:54.254796 10136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:54.255048 10136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:54.255084 10136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:54.258852 10136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:54.258894 10136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:54.258910 10136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:54.258926 10136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:54.258940 10136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:54.263528 10136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:54.264441 10136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:54.265771 10136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:54.274132 10144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:54.274113 10141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:54.274278 10142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:54.274391 10136 server_base.cc:1034] running on GCE node
I20260926 07:25:54.274904 10136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:54.275986 10136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:54.277172 10136 hybrid_clock.cc:648] HybridClock initialized: now 1790407554277123 us; error 68 us; skew 500 ppm
I20260926 07:25:54.278920 10136 webserver.cc:460] Webserver started at http://127.0.98.62:40545/ using document root <none> and password file <none>
I20260926 07:25:54.279528 10136 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:54.279618 10136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:54.279855 10136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:54.281836 10136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "5301c6bd83014533840bb8f814f43d8b"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "ba50c87ecce06a9cab89f9b79b429518"
server_key_iv: "2a0b90c32ef8bc9b4d2cdc77ff4bf724"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.282379 10136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "5301c6bd83014533840bb8f814f43d8b"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "ba50c87ecce06a9cab89f9b79b429518"
server_key_iv: "2a0b90c32ef8bc9b4d2cdc77ff4bf724"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.285832 10136 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:25:54.288335 10150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:54.289592 10136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:54.289729 10136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "5301c6bd83014533840bb8f814f43d8b"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "ba50c87ecce06a9cab89f9b79b429518"
server_key_iv: "2a0b90c32ef8bc9b4d2cdc77ff4bf724"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.289837 10136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:54.294278 10136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:54.294832 10136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:54.295014 10136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:54.301612 10136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43105
I20260926 07:25:54.301620 10202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43105 every 8 connection(s)
I20260926 07:25:54.302443 10136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:54.305410 10203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:54.310575 10203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b: Bootstrap starting.
I20260926 07:25:54.311137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 10136
I20260926 07:25:54.311259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:54.311566 392 external_mini_cluster.cc:1428] Setting key 907ae254e6ca40b681a3d39db168bf32
I20260926 07:25:54.313248 10203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:54.313989 10203 log.cc:824] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:54.316269 10203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b: No bootstrap required, opened a new log
I20260926 07:25:54.317059 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:54.312860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.0.98.62:43105)
0926 07:25:54.313490 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:54.313505 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:54.313540 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:25:54.314171 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:54.314193 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:54.314603 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:54.314929 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:54.315778 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:54.315783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:54.315794 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:54.315803 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:54.315854 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:54.315869 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:54.315963 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:54.316097 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 07:25:54.316371 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":126,"threads_started":1}
I20260926 07:25:54.320101 10203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5301c6bd83014533840bb8f814f43d8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } }
I20260926 07:25:54.320338 10203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:54.320417 10203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5301c6bd83014533840bb8f814f43d8b, State: Initialized, Role: FOLLOWER
I20260926 07:25:54.320830 10203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5301c6bd83014533840bb8f814f43d8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } }
I20260926 07:25:54.320977 10203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:54.321064 10203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:54.321169 10203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:54.322045 10203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5301c6bd83014533840bb8f814f43d8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } }
I20260926 07:25:54.322378 10203 leader_election.cc:302] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5301c6bd83014533840bb8f814f43d8b; no voters:
I20260926 07:25:54.322676 10203 leader_election.cc:288] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:54.322753 10208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:54.323007 10208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [term 1 LEADER]: Becoming Leader. State: Replica: 5301c6bd83014533840bb8f814f43d8b, State: Running, Role: LEADER
I20260926 07:25:54.323356 10208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5301c6bd83014533840bb8f814f43d8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } }
I20260926 07:25:54.323909 10203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:54.324427 10208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5301c6bd83014533840bb8f814f43d8b. Latest consensus state: current_term: 1 leader_uuid: "5301c6bd83014533840bb8f814f43d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5301c6bd83014533840bb8f814f43d8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } }
I20260926 07:25:54.324555 10208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:54.325090 10210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5301c6bd83014533840bb8f814f43d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5301c6bd83014533840bb8f814f43d8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } }
I20260926 07:25:54.325193 10210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:54.325510 10213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:54.328125 10213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:54.333634 10213 catalog_manager.cc:1347] Generated new cluster ID: 224fadee2fb542c784af2eb66f963730
I20260926 07:25:54.333698 10213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:54.355134 10213 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:54.355459 10213 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:54.369306 10213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5301c6bd83014533840bb8f814f43d8b: Generated new TSK 0
I20260926 07:25:54.370097 10213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:25:54.383996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43105
--builtin_ntp_servers=127.0.98.20:36849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:25:54.494020 10227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:54.494328 10227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:54.494412 10227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:54.498418 10227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:54.498497 10227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:54.498587 10227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:54.503319 10227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:54.504370 10227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:54.505751 10227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:54.515223 10235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:54.515223 10233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:54.515283 10232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:54.515581 10227 server_base.cc:1034] running on GCE node
I20260926 07:25:54.515969 10227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:54.516602 10227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:54.517781 10227 hybrid_clock.cc:648] HybridClock initialized: now 1790407554517761 us; error 30 us; skew 500 ppm
I20260926 07:25:54.519757 10227 webserver.cc:460] Webserver started at http://127.0.98.1:37319/ using document root <none> and password file <none>
I20260926 07:25:54.520314 10227 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:54.520406 10227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:54.520624 10227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:54.522605 10227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "0778a7bff6b5434b834821c078646cb4"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "d6e803d79667bf33dc5acbcb465db980"
server_key_iv: "b93e6218a6cd402c297cc1bd74164fee"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.523212 10227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0778a7bff6b5434b834821c078646cb4"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "d6e803d79667bf33dc5acbcb465db980"
server_key_iv: "b93e6218a6cd402c297cc1bd74164fee"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.526978 10227 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:54.529265 10241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:54.530504 10227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:25:54.530593 10227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "0778a7bff6b5434b834821c078646cb4"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "d6e803d79667bf33dc5acbcb465db980"
server_key_iv: "b93e6218a6cd402c297cc1bd74164fee"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.530671 10227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:54.534878 10227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:54.535454 10227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:54.535588 10227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:54.536048 10227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:54.536828 10227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:54.536873 10227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:54.536908 10227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:54.536932 10227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:54.545797 10227 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35815
I20260926 07:25:54.545822 10354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35815 every 8 connection(s)
I20260926 07:25:54.546720 10227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:54.550014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 10227
I20260926 07:25:54.550127 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:25:54.550439 392 external_mini_cluster.cc:1428] Setting key fcc229fdbc4d9519f670e1e16c7793aa
I20260926 07:25:54.552620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43105
--builtin_ntp_servers=127.0.98.20:36849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 07:25:54.552948 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:54.548535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57357 (local address 127.0.98.62:43105)
0926 07:25:54.548666 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:54.548670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:54.549687 (+ 1017us) server_negotiation.cc:407] Connection header received
0926 07:25:54.551155 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:54.551159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:54.551200 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:54.551299 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:54.552482 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:54.552487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:54.552490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:54.552506 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:54.552530 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:54.552541 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:54.552634 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:54.552782 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 07:25:54.552823 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:25:54.553747 10357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:54.548930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43105 (local address 127.0.98.1:57357)
0926 07:25:54.549529 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:54.549568 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:54.550874 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:54.551475 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:54.551484 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:54.551588 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:54.552227 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:54.552229 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:54.552268 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:54.552331 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:54.552628 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:54.552633 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:54.552637 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:25:54.552795 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 07:25:54.553030 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":100,"threads_started":1}
I20260926 07:25:54.554891 10355 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43105
I20260926 07:25:54.555128 10355 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:54.555508 10355 heartbeater.cc:511] Master 127.0.98.62:43105 requested a full tablet report, sending...
I20260926 07:25:54.557085 10167 ts_manager.cc:194] Registered new tserver with Master: 0778a7bff6b5434b834821c078646cb4 (127.0.98.1:35815)
W20260926 07:25:54.661489 10358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:54.661751 10358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:54.661790 10358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:54.665756 10358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:54.665817 10358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:54.665871 10358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:54.670431 10358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:54.671548 10358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:54.672881 10358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:54.682432 10366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:54.682534 10364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:54.682610 10358 server_base.cc:1034] running on GCE node
W20260926 07:25:54.682432 10363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:54.683193 10358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:54.683809 10358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:54.684983 10358 hybrid_clock.cc:648] HybridClock initialized: now 1790407554684969 us; error 30 us; skew 500 ppm
I20260926 07:25:54.686844 10358 webserver.cc:460] Webserver started at http://127.0.98.2:42309/ using document root <none> and password file <none>
I20260926 07:25:54.687439 10358 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:54.687500 10358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:54.687659 10358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:54.689652 10358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "e05191143bfb4a06a30c4a5ba8e89c2e"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "3847e276a97df8e79bec5fe5a3e40930"
server_key_iv: "518cff077115c38f7a3e8053f4479f51"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.690316 10358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e05191143bfb4a06a30c4a5ba8e89c2e"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "3847e276a97df8e79bec5fe5a3e40930"
server_key_iv: "518cff077115c38f7a3e8053f4479f51"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.693933 10358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:25:54.696393 10372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:54.697650 10358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:54.697782 10358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "e05191143bfb4a06a30c4a5ba8e89c2e"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "3847e276a97df8e79bec5fe5a3e40930"
server_key_iv: "518cff077115c38f7a3e8053f4479f51"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.697894 10358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:54.702420 10358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:54.703003 10358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:54.703212 10358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:54.703752 10358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:54.704653 10358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:54.704732 10358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:54.704797 10358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:54.704851 10358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:54.713136 10358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35599
I20260926 07:25:54.713140 10485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35599 every 8 connection(s)
I20260926 07:25:54.713948 10358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:25:54.718606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 10358
I20260926 07:25:54.718736 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:25:54.719133 392 external_mini_cluster.cc:1428] Setting key 126dc85c8357d2cdb1c675cf89ce231a
I20260926 07:25:54.720722 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:54.715628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58253 (local address 127.0.98.62:43105)
0926 07:25:54.715789 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:54.715794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:54.716872 (+ 1078us) server_negotiation.cc:407] Connection header received
0926 07:25:54.718227 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:54.718230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:54.718265 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:54.718389 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:54.719815 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:54.719818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:54.719820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:54.719831 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:54.719850 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:54.719858 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:54.719936 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:54.720561 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 07:25:54.720605 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:25:54.721267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43105
--builtin_ntp_servers=127.0.98.20:36849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 07:25:54.721767 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:54.716049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43105 (local address 127.0.98.2:58253)
0926 07:25:54.716713 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:54.716765 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:54.718012 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:54.718605 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:54.718615 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:54.718754 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:54.719536 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:54.719537 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:54.719574 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:54.719645 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:54.720388 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:54.720398 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:54.720404 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:25:54.720551 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 07:25:54.720791 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":94,"threads_started":1}
I20260926 07:25:54.722916 10486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43105
I20260926 07:25:54.723189 10486 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:54.723603 10486 heartbeater.cc:511] Master 127.0.98.62:43105 requested a full tablet report, sending...
I20260926 07:25:54.724742 10167 ts_manager.cc:194] Registered new tserver with Master: e05191143bfb4a06a30c4a5ba8e89c2e (127.0.98.2:35599)
W20260926 07:25:54.833709 10489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:54.834007 10489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:54.834086 10489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:54.838199 10489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:54.838282 10489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:54.838370 10489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:54.843267 10489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:54.844403 10489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:54.845826 10489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:54.855312 10495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:54.855355 10497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:54.855400 10494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:54.855720 10489 server_base.cc:1034] running on GCE node
I20260926 07:25:54.856150 10489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:54.856808 10489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:54.858014 10489 hybrid_clock.cc:648] HybridClock initialized: now 1790407554857960 us; error 35 us; skew 500 ppm
I20260926 07:25:54.859947 10489 webserver.cc:460] Webserver started at http://127.0.98.3:32981/ using document root <none> and password file <none>
I20260926 07:25:54.860550 10489 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:54.860617 10489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:54.860870 10489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:54.863013 10489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "7cfb7e96ed184de7a037e9c4b9980a0a"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "f7bfe4040836304a37ac12fdf8c53f2b"
server_key_iv: "2b3df3cad24a822af097119763aaf2e9"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.863669 10489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7cfb7e96ed184de7a037e9c4b9980a0a"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "f7bfe4040836304a37ac12fdf8c53f2b"
server_key_iv: "2b3df3cad24a822af097119763aaf2e9"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.867542 10489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 07:25:54.870182 10503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:54.871601 10489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:54.871729 10489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "7cfb7e96ed184de7a037e9c4b9980a0a"
format_stamp: "Formatted at 2026-09-26 07:25:54 on dist-test-slave-g04g"
server_key: "f7bfe4040836304a37ac12fdf8c53f2b"
server_key_iv: "2b3df3cad24a822af097119763aaf2e9"
server_key_version: "encryptionkey@0"
I20260926 07:25:54.871836 10489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:54.876189 10489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:54.876734 10489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:54.876912 10489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:54.877445 10489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:54.878319 10489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:54.878384 10489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:54.878444 10489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:54.878487 10489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:54.887002 10489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42221
I20260926 07:25:54.887050 10616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42221 every 8 connection(s)
I20260926 07:25:54.887913 10489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:54.894241 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:54.889785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55223 (local address 127.0.98.62:43105)
0926 07:25:54.889971 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:54.889976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:54.891287 (+ 1311us) server_negotiation.cc:407] Connection header received
0926 07:25:54.892627 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:54.892630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:54.892668 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:54.892757 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:54.893808 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:54.893811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:54.893813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:25:54.893826 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:54.893847 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:25:54.893856 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:54.893954 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:54.894046 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 07:25:54.894106 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 07:25:54.895175 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:54.890306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43105 (local address 127.0.98.3:55223)
0926 07:25:54.891098 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:54.891150 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:54.892414 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:54.892894 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:54.892901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:54.892995 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:54.893537 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:25:54.893538 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:54.893578 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:25:54.893645 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:54.893939 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:54.893943 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:54.893947 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:25:54.894063 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 07:25:54.894356 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":142,"threads_started":1}
I20260926 07:25:54.896412 10617 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43105
I20260926 07:25:54.896658 10617 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:54.897050 10617 heartbeater.cc:511] Master 127.0.98.62:43105 requested a full tablet report, sending...
I20260926 07:25:54.897941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 10489
I20260926 07:25:54.898025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:54.898185 10167 ts_manager.cc:194] Registered new tserver with Master: 7cfb7e96ed184de7a037e9c4b9980a0a (127.0.98.3:42221)
I20260926 07:25:54.898343 392 external_mini_cluster.cc:1428] Setting key dd95ce2e221c1a601d8638d7d2ef1501
I20260926 07:25:54.900264 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:25:54.903138 10206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:25:54.902178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33032 (local address 127.0.98.62:43105)
0926 07:25:54.902288 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:54.902294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:54.902531 (+ 237us) server_negotiation.cc:407] Connection header received
0926 07:25:54.902662 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:54.902665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:54.902696 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:54.902782 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:54.902981 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33032: BlockingRecv error: recv got EOF from 127.0.0.1:33032 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:25:54.903458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 10227
I20260926 07:25:54.908908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 10358
I20260926 07:25:54.914165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 10489
I20260926 07:25:54.919395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 10136
2026-09-26T07:25:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (801 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:25:54 dist-test-slave-g04g krb5kdc[10627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:25:54 dist-test-slave-g04g krb5kdc[10627](info): set up 2 sockets
Sep 26 07:25:54 dist-test-slave-g04g krb5kdc[10627](info): commencing operation
krb5kdc: starting...
W20260926 07:25:55.960086 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:25:56 dist-test-slave-g04g krb5kdc[10627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:25:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:25:56.079079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33235
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:56.188234 10640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:56.188516 10640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:56.188587 10640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:56.192550 10640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:25:56.192631 10640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:56.192665 10640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:56.192724 10640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:25:56.192765 10640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:25:56.197912 10640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:56.198943 10640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:56.199961 10640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:56.208649 10646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:56.208657 10645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:56.208660 10648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:56.208874 10640 server_base.cc:1034] running on GCE node
I20260926 07:25:56.209442 10640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:56.210517 10640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:56.211701 10640 hybrid_clock.cc:648] HybridClock initialized: now 1790407556211677 us; error 49 us; skew 500 ppm
Sep 26 07:25:56 dist-test-slave-g04g krb5kdc[10627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:56.220077 10640 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:25:56.221022 10640 webserver.cc:460] Webserver started at http://127.0.98.62:45055/ using document root <none> and password file <none>
I20260926 07:25:56.221575 10640 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:56.221662 10640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:56.221894 10640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:56.224041 10640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "b3467fc7cb38461790bdb1d9a7c7a925"
format_stamp: "Formatted at 2026-09-26 07:25:56 on dist-test-slave-g04g"
server_key: "858212bf7d8553fb2964be90e7136252"
server_key_iv: "14aeae47ba3c9ee01eb85d577afd3d5f"
server_key_version: "encryptionkey@0"
I20260926 07:25:56.224597 10640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "b3467fc7cb38461790bdb1d9a7c7a925"
format_stamp: "Formatted at 2026-09-26 07:25:56 on dist-test-slave-g04g"
server_key: "858212bf7d8553fb2964be90e7136252"
server_key_iv: "14aeae47ba3c9ee01eb85d577afd3d5f"
server_key_version: "encryptionkey@0"
I20260926 07:25:56.229099 10640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:25:56.231812 10655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:56.232930 10640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:56.233073 10640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "b3467fc7cb38461790bdb1d9a7c7a925"
format_stamp: "Formatted at 2026-09-26 07:25:56 on dist-test-slave-g04g"
server_key: "858212bf7d8553fb2964be90e7136252"
server_key_iv: "14aeae47ba3c9ee01eb85d577afd3d5f"
server_key_version: "encryptionkey@0"
I20260926 07:25:56.233179 10640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:56.247432 10640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:56.250223 10640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:56.250427 10640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:56.257557 10640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33235
I20260926 07:25:56.257865 10707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33235 every 8 connection(s)
I20260926 07:25:56.258486 10640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:25:56.261224 10708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:56.264714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 10640
I20260926 07:25:56.264873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:25:56.265158 392 external_mini_cluster.cc:1428] Setting key afa8389557af79d1034e94bacd394878
I20260926 07:25:56.266875 10708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925: Bootstrap starting.
I20260926 07:25:56.269625 10708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:56.270496 10708 log.cc:824] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:56.272552 10708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925: No bootstrap required, opened a new log
Sep 26 07:25:56 dist-test-slave-g04g krb5kdc[10627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:56.277189 10708 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3467fc7cb38461790bdb1d9a7c7a925" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } }
I20260926 07:25:56.277462 10708 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:56.277536 10708 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3467fc7cb38461790bdb1d9a7c7a925, State: Initialized, Role: FOLLOWER
I20260926 07:25:56.278030 10708 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3467fc7cb38461790bdb1d9a7c7a925" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } }
I20260926 07:25:56.278203 10708 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:25:56.278286 10708 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:25:56.278398 10708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:56.279501 10708 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3467fc7cb38461790bdb1d9a7c7a925" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } }
I20260926 07:25:56.279856 10708 leader_election.cc:302] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3467fc7cb38461790bdb1d9a7c7a925; no voters:
I20260926 07:25:56.280174 10708 leader_election.cc:288] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:25:56.280303 10713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:56.280555 10713 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [term 1 LEADER]: Becoming Leader. State: Replica: b3467fc7cb38461790bdb1d9a7c7a925, State: Running, Role: LEADER
I20260926 07:25:56.280839 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:56.266467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.0.98.62:33235)
0926 07:25:56.266877 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:56.266899 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:56.267082 (+ 183us) server_negotiation.cc:407] Connection header received
0926 07:25:56.267943 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:56.267965 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:56.268389 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:56.268740 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:56.269350 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.270865 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.271897 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.272603 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.272782 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:56.275470 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:56.275480 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:56.275502 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:56.275544 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:56.277589 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:56.278039 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:56.278048 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:56.278055 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:56.278204 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:56.278576 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:56.278581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:56.278586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:56.279123 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:56.279334 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:56.279542 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 07:25:56.279829 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":111,"threads_started":1}
I20260926 07:25:56.280877 10713 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3467fc7cb38461790bdb1d9a7c7a925" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } }
I20260926 07:25:56.281445 10708 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:25:56.282555 10714 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3467fc7cb38461790bdb1d9a7c7a925. Latest consensus state: current_term: 1 leader_uuid: "b3467fc7cb38461790bdb1d9a7c7a925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3467fc7cb38461790bdb1d9a7c7a925" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } } }
I20260926 07:25:56.282673 10714 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:56.283165 10722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:25:56.283779 10715 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3467fc7cb38461790bdb1d9a7c7a925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3467fc7cb38461790bdb1d9a7c7a925" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } } }
I20260926 07:25:56.283869 10715 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925 [sys.catalog]: This master's current role is: LEADER
I20260926 07:25:56.286048 10722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:25:56.291257 10722 catalog_manager.cc:1347] Generated new cluster ID: 484fd2bddd7a438ba22e4382ccc27814
I20260926 07:25:56.291323 10722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:25:56.299232 10722 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:25:56.300128 10722 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:25:56.309960 10722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925: Generated new TSK 0
I20260926 07:25:56.310571 10722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:25:56.337896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33235
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:56.446589 10734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:56.446828 10734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:56.446897 10734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:56.450855 10734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:56.450909 10734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:56.451000 10734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:25:56.456130 10734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:56.457028 10734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:56.457947 10734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:56.467459 10740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:56.467459 10742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:56.467468 10739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:56.467831 10734 server_base.cc:1034] running on GCE node
I20260926 07:25:56.468256 10734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:56.468907 10734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:56.470090 10734 hybrid_clock.cc:648] HybridClock initialized: now 1790407556470067 us; error 36 us; skew 500 ppm
Sep 26 07:25:56 dist-test-slave-g04g krb5kdc[10627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:56.472769 10734 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:25:56.473579 10734 webserver.cc:460] Webserver started at http://127.0.98.1:40927/ using document root <none> and password file <none>
I20260926 07:25:56.474109 10734 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:56.474203 10734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:56.474421 10734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:56.476369 10734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "1efe422d7074432788315ca5ecce415f"
format_stamp: "Formatted at 2026-09-26 07:25:56 on dist-test-slave-g04g"
server_key: "9b4f78c74af197d9321b0e4b0ed3aaa5"
server_key_iv: "96a60c58937fc619a9c904eb2596f8b0"
server_key_version: "encryptionkey@0"
I20260926 07:25:56.476912 10734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1efe422d7074432788315ca5ecce415f"
format_stamp: "Formatted at 2026-09-26 07:25:56 on dist-test-slave-g04g"
server_key: "9b4f78c74af197d9321b0e4b0ed3aaa5"
server_key_iv: "96a60c58937fc619a9c904eb2596f8b0"
server_key_version: "encryptionkey@0"
I20260926 07:25:56.480433 10734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:25:56.482765 10749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:56.484051 10734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 07:25:56.484220 10734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "1efe422d7074432788315ca5ecce415f"
format_stamp: "Formatted at 2026-09-26 07:25:56 on dist-test-slave-g04g"
server_key: "9b4f78c74af197d9321b0e4b0ed3aaa5"
server_key_iv: "96a60c58937fc619a9c904eb2596f8b0"
server_key_version: "encryptionkey@0"
I20260926 07:25:56.484330 10734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:56.493688 10734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:56.496415 10734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:56.496603 10734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:56.497120 10734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:56.497891 10734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:56.497960 10734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:56.498030 10734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:56.498082 10734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:56.506964 10734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39639
I20260926 07:25:56.506987 10862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39639 every 8 connection(s)
I20260926 07:25:56.507876 10734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:25:56.513724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 10734
I20260926 07:25:56.513845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:25:56.514127 392 external_mini_cluster.cc:1428] Setting key b16552ed60dbbdf31831246124f9808f
Sep 26 07:25:56 dist-test-slave-g04g krb5kdc[10627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:56.521956 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:56.509657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47959 (local address 127.0.98.62:33235)
0926 07:25:56.509787 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:56.509790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:56.510795 (+ 1005us) server_negotiation.cc:407] Connection header received
0926 07:25:56.511714 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:56.511717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:56.511773 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:56.511888 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:56.513011 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.513667 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.514979 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.515391 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.515500 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:56.518459 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:56.518463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:56.518465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:56.518487 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:56.519956 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:56.520460 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:56.520465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:56.520467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:56.520546 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:56.520890 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:56.520893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:56.520895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:56.521074 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:56.521180 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:56.521666 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 07:25:56.521793 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:25:56.522763 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:56.510020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33235 (local address 127.0.98.1:47959)
0926 07:25:56.510638 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:56.510677 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:56.511515 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:56.512053 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:56.512059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:56.512416 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:56.512871 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:56.512883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.513905 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.513909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:56.514866 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:56.514876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.515565 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.515569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:56.515758 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:56.516344 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:56.516370 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:56.518285 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:56.520076 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:56.520082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:56.520097 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:56.520357 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:56.520644 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:56.520647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:56.520649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:56.520786 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:56.521179 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:56.521195 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:56.521441 (+ 246us) client_negotiation.cc:769] Sending connection context
0926 07:25:56.521628 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 07:25:56.521941 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 07:25:56.524036 10863 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33235
I20260926 07:25:56.524240 10863 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:56.524724 10863 heartbeater.cc:511] Master 127.0.98.62:33235 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 07:25:56.526405 10667 ts_manager.cc:194] Registered new tserver with Master: 1efe422d7074432788315ca5ecce415f (127.0.98.1:39639)
I20260926 07:25:56.527858 10667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:25:56.542446 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33235
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:56.655524 10868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:56.655777 10868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:56.655817 10868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:56.659816 10868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:56.659871 10868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:56.659927 10868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:25:56.665021 10868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:56.665957 10868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:56.666895 10868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:56.676853 10873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:56.676988 10874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:56.677027 10868 server_base.cc:1034] running on GCE node
W20260926 07:25:56.676988 10876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:56.677506 10868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:56.678171 10868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:56.679355 10868 hybrid_clock.cc:648] HybridClock initialized: now 1790407556679340 us; error 29 us; skew 500 ppm
Sep 26 07:25:56 dist-test-slave-g04g krb5kdc[10627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:56.682051 10868 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:25:56.682999 10868 webserver.cc:460] Webserver started at http://127.0.98.2:41863/ using document root <none> and password file <none>
I20260926 07:25:56.683648 10868 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:56.683753 10868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:56.683969 10868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:56.685981 10868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "76ffb1129a0641fd91d08eb80e96c5ac"
format_stamp: "Formatted at 2026-09-26 07:25:56 on dist-test-slave-g04g"
server_key: "a535804f4cb3ea22da5717884c40bfb0"
server_key_iv: "a136477c25a24ae1e051d577b150c706"
server_key_version: "encryptionkey@0"
I20260926 07:25:56.686553 10868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "76ffb1129a0641fd91d08eb80e96c5ac"
format_stamp: "Formatted at 2026-09-26 07:25:56 on dist-test-slave-g04g"
server_key: "a535804f4cb3ea22da5717884c40bfb0"
server_key_iv: "a136477c25a24ae1e051d577b150c706"
server_key_version: "encryptionkey@0"
I20260926 07:25:56.690280 10868 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:25:56.692664 10883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:56.693929 10868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:25:56.694064 10868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "76ffb1129a0641fd91d08eb80e96c5ac"
format_stamp: "Formatted at 2026-09-26 07:25:56 on dist-test-slave-g04g"
server_key: "a535804f4cb3ea22da5717884c40bfb0"
server_key_iv: "a136477c25a24ae1e051d577b150c706"
server_key_version: "encryptionkey@0"
I20260926 07:25:56.694175 10868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:56.715534 10868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:56.718353 10868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:56.718554 10868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:56.719161 10868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:56.719972 10868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:56.720042 10868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:56.720111 10868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:56.720161 10868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:56.728729 10868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44429
I20260926 07:25:56.728749 10996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44429 every 8 connection(s)
I20260926 07:25:56.729579 10868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:25:56 dist-test-slave-g04g krb5kdc[10627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:56.738415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 10868
I20260926 07:25:56.738546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:25:56.738830 392 external_mini_cluster.cc:1428] Setting key 8f1faa656699c008f07d3da2666a959a
I20260926 07:25:56.742552 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:56.731242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52833 (local address 127.0.98.62:33235)
0926 07:25:56.731400 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:56.731405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:56.732457 (+ 1052us) server_negotiation.cc:407] Connection header received
0926 07:25:56.733274 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:56.733277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:56.733321 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:56.733394 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:56.734478 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.734988 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.736145 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.736383 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.736459 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:56.738893 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:56.738897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:56.738901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:56.738930 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:56.740454 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:56.740941 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:56.740946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:56.740948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:56.741032 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:56.741368 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:56.741373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:56.741375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:56.741553 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:56.741672 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:56.742238 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 07:25:56.742366 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:25:56.743340 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:56.731680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33235 (local address 127.0.98.2:52833)
0926 07:25:56.732310 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:56.732349 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:56.733076 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:56.733552 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:56.733559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:56.733893 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:56.734354 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:56.734367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.735145 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.735148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:56.736003 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:56.736011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.736499 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.736502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:56.736645 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:56.737227 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:56.737243 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:56.738738 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:56.740565 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:56.740571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:56.740584 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:56.740838 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:56.741128 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:56.741131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:56.741133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:56.741272 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:56.741700 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:56.741720 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:56.741992 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 07:25:56.742150 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 07:25:56.742380 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":109,"threads_started":1}
I20260926 07:25:56.744432 10997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33235
I20260926 07:25:56.744663 10997 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:56.745118 10997 heartbeater.cc:511] Master 127.0.98.62:33235 requested a full tablet report, sending...
I20260926 07:25:56.746189 10667 ts_manager.cc:194] Registered new tserver with Master: 76ffb1129a0641fd91d08eb80e96c5ac (127.0.98.2:44429)
I20260926 07:25:56.747071 10667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52833
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:25:56.766396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33235
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:25:56.877336 11002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:25:56.877578 11002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:25:56.877620 11002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:25:56.881700 11002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:25:56.881752 11002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:25:56.881872 11002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:25:56.887110 11002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:25:56.888106 11002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:25:56.889106 11002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:25:56.898767 11007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:56.898769 11010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:25:56.898769 11008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:25:56.899021 11002 server_base.cc:1034] running on GCE node
I20260926 07:25:56.899537 11002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:25:56.900154 11002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:25:56.901336 11002 hybrid_clock.cc:648] HybridClock initialized: now 1790407556901318 us; error 31 us; skew 500 ppm
Sep 26 07:25:56 dist-test-slave-g04g krb5kdc[10627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:25:56.904075 11002 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:25:56.904907 11002 webserver.cc:460] Webserver started at http://127.0.98.3:37757/ using document root <none> and password file <none>
I20260926 07:25:56.905440 11002 fs_manager.cc:359] Metadata directory not provided
I20260926 07:25:56.905525 11002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:25:56.905764 11002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:25:56.907763 11002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "2f37d9f42ab24153831e7a1078b56049"
format_stamp: "Formatted at 2026-09-26 07:25:56 on dist-test-slave-g04g"
server_key: "6b1aaab1dd9fa0977868c87fd78a6501"
server_key_iv: "4fb7bc3d6122d7384844849f61d2c06c"
server_key_version: "encryptionkey@0"
I20260926 07:25:56.908296 11002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2f37d9f42ab24153831e7a1078b56049"
format_stamp: "Formatted at 2026-09-26 07:25:56 on dist-test-slave-g04g"
server_key: "6b1aaab1dd9fa0977868c87fd78a6501"
server_key_iv: "4fb7bc3d6122d7384844849f61d2c06c"
server_key_version: "encryptionkey@0"
I20260926 07:25:56.911792 11002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:25:56.914034 11017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:56.915278 11002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:25:56.915396 11002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "2f37d9f42ab24153831e7a1078b56049"
format_stamp: "Formatted at 2026-09-26 07:25:56 on dist-test-slave-g04g"
server_key: "6b1aaab1dd9fa0977868c87fd78a6501"
server_key_iv: "4fb7bc3d6122d7384844849f61d2c06c"
server_key_version: "encryptionkey@0"
I20260926 07:25:56.915495 11002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:25:56.940927 11002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:25:56.944274 11002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:25:56.944480 11002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:25:56.945096 11002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:25:56.945911 11002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:25:56.945982 11002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:56.946050 11002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:25:56.946095 11002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:25:56.954572 11002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43205
I20260926 07:25:56.954594 11130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43205 every 8 connection(s)
I20260926 07:25:56.955505 11002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:25:56.962338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 11002
I20260926 07:25:56.962451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:25:56.962740 392 external_mini_cluster.cc:1428] Setting key 4130809bf7b58abd5242e255fda04f2b
Sep 26 07:25:56 dist-test-slave-g04g krb5kdc[10627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:25:56.968780 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:56.957353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46445 (local address 127.0.98.62:33235)
0926 07:25:56.957506 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:56.957510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:56.958553 (+ 1043us) server_negotiation.cc:407] Connection header received
0926 07:25:56.959483 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:56.959485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:56.959533 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:56.959609 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:56.960723 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.961238 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.962392 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.962721 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.962833 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:56.965549 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:56.965552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:56.965554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:56.965574 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:56.967080 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:56.967517 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:56.967519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:56.967520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:56.967589 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:56.967888 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:56.967891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:56.967892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:56.968052 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:56.968135 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:56.968561 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 07:25:56.968668 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:25:56.969606 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:56.957724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33235 (local address 127.0.98.3:46445)
0926 07:25:56.958397 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:56.958443 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:56.959304 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:56.959764 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:56.959770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:56.960141 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:25:56.960603 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:56.960616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.961346 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.961348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:56.962258 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:56.962266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.962910 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.962914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:56.963140 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:56.963771 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:25:56.963790 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:25:56.965410 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:25:56.967179 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:56.967183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:56.967194 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:56.967435 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:56.967679 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:25:56.967681 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:25:56.967682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:25:56.967811 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:25:56.968135 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:25:56.968140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:25:56.968366 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 07:25:56.968518 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 07:25:56.968760 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":111,"threads_started":1}
I20260926 07:25:56.970713 11131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33235
I20260926 07:25:56.970948 11131 heartbeater.cc:464] Registering TS with master...
I20260926 07:25:56.971436 11131 heartbeater.cc:511] Master 127.0.98.62:33235 requested a full tablet report, sending...
I20260926 07:25:56.972432 10667 ts_manager.cc:194] Registered new tserver with Master: 2f37d9f42ab24153831e7a1078b56049 (127.0.98.3:43205)
I20260926 07:25:56.972983 10667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46445
I20260926 07:25:56.976527 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:25:56.985575 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:56.978297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.0.98.62:33235)
0926 07:25:56.978426 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:56.978431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:56.978573 (+ 142us) server_negotiation.cc:407] Connection header received
0926 07:25:56.978669 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:56.978671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:56.978707 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:56.978783 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:25:56.979478 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.979974 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.980880 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:56.981076 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:56.981149 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:56.982684 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:25:56.982720 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:25:56.982722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:25:56.982763 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:25:56.984184 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:56.984582 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:56.984584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:56.984586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:56.984656 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:25:56.984901 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:25:56.984903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:25:56.984904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:25:56.985093 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:25:56.985191 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:56.985380 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:25:56.985467 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:25:56.988446 10667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:25:56.991521 10667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:25:57.005499 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:56.999659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39639 (local address 127.0.0.1:55516)
0926 07:25:57.000002 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:57.000052 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:57.000203 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:57.001279 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:57.001286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:57.001320 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:57.001481 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:57.001490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.002831 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.002835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:57.003791 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:57.003800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.004923 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.004926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:57.005047 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:57.005083 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:25:57.005196 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 07:25:57.005329 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":28,"spinlock_wait_cycles":5248,"thread_start_us":81,"threads_started":1}
I20260926 07:25:57.005548 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:57.000376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44429 (local address 127.0.0.1:34902)
0926 07:25:57.000623 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:57.000637 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:57.000751 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:57.001615 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:57.001620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:57.001636 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:57.001714 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:57.001720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.002708 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.002713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:57.004150 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:57.004158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.005224 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.005227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:57.005311 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:57.005330 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 07:25:57.005388 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 07:25:57.005444 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":33,"thread_start_us":125,"threads_started":1}
I20260926 07:25:57.005874 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:57.000820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.0.98.2:44429)
0926 07:25:57.001134 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:57.001145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:57.001168 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:25:57.001243 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:57.001248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:57.001449 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:57.001646 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:57.001824 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.002574 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.004275 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.005113 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.005225 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:57.005356 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:57.005418 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 07:25:57.005736 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":140,"threads_started":1}
I20260926 07:25:57.006688 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:57.000035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43205 (local address 127.0.0.1:49988)
0926 07:25:57.000295 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:57.000308 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:57.000383 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:57.001279 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:57.001286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:57.001322 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:57.001642 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:57.001648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.004150 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.004153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:57.005296 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:57.005306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.006326 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.006330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:57.006413 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:57.006433 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:25:57.006495 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 07:25:57.006559 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":213,"spinlock_wait_cycles":6016,"thread_start_us":97,"threads_started":1}
I20260926 07:25:57.006755 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:57.000449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.0.98.3:43205)
0926 07:25:57.000761 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:57.000773 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:57.000797 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:25:57.000871 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:57.000877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:57.001085 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:57.001429 (+ 344us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:57.002835 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.003997 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.005444 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.006200 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.006327 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:57.006461 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:57.006518 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 07:25:57.006615 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":135,"threads_started":1}
I20260926 07:25:57.007102 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:56.999814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.0.98.1:39639)
0926 07:25:57.000895 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:57.000909 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:57.000927 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:25:57.000991 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:57.000995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:57.001134 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:57.001804 (+ 670us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:57.001941 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.002669 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.003962 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.004797 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.004925 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:57.005081 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:57.006812 (+ 1731us) server_negotiation.cc:299] Negotiation successful
0926 07:25:57.006927 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":985,"thread_start_us":114,"threads_started":1}
I20260926 07:25:57.009069 10931 tablet_service.cc:1467] Processing CreateTablet for tablet 862ac19de76e495bbc62c3f0a57be726 (DEFAULT_TABLE table=test-table [id=9f16adbdaf884372ba4756592ac3c7d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:57.009311 11065 tablet_service.cc:1467] Processing CreateTablet for tablet 862ac19de76e495bbc62c3f0a57be726 (DEFAULT_TABLE table=test-table [id=9f16adbdaf884372ba4756592ac3c7d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:57.009603 10797 tablet_service.cc:1467] Processing CreateTablet for tablet 862ac19de76e495bbc62c3f0a57be726 (DEFAULT_TABLE table=test-table [id=9f16adbdaf884372ba4756592ac3c7d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:25:57.010236 11065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862ac19de76e495bbc62c3f0a57be726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:57.010232 10931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862ac19de76e495bbc62c3f0a57be726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:57.010494 10797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 862ac19de76e495bbc62c3f0a57be726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:25:57.016848 11148 tablet_bootstrap.cc:492] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049: Bootstrap starting.
I20260926 07:25:57.018953 11148 tablet_bootstrap.cc:654] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:57.019248 11149 tablet_bootstrap.cc:492] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f: Bootstrap starting.
I20260926 07:25:57.019445 11150 tablet_bootstrap.cc:492] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac: Bootstrap starting.
I20260926 07:25:57.020262 11148 log.cc:824] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:57.021283 11149 tablet_bootstrap.cc:654] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:57.021855 11150 tablet_bootstrap.cc:654] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac: Neither blocks nor log segments found. Creating new log.
I20260926 07:25:57.022540 11149 log.cc:824] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:57.022646 11150 log.cc:824] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac: Log is configured to *not* fsync() on all Append() calls
I20260926 07:25:57.022995 11148 tablet_bootstrap.cc:492] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049: No bootstrap required, opened a new log
I20260926 07:25:57.023207 11148 ts_tablet_manager.cc:1397] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 07:25:57.024186 11149 tablet_bootstrap.cc:492] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f: No bootstrap required, opened a new log
I20260926 07:25:57.024382 11149 ts_tablet_manager.cc:1397] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:25:57.026186 11148 raft_consensus.cc:348] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe422d7074432788315ca5ecce415f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39639 } } peers { permanent_uuid: "76ffb1129a0641fd91d08eb80e96c5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44429 } } peers { permanent_uuid: "2f37d9f42ab24153831e7a1078b56049" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43205 } }
I20260926 07:25:57.026423 11148 raft_consensus.cc:374] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:57.026479 11148 raft_consensus.cc:729] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f37d9f42ab24153831e7a1078b56049, State: Initialized, Role: FOLLOWER
I20260926 07:25:57.026853 11148 consensus_queue.cc:260] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe422d7074432788315ca5ecce415f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39639 } } peers { permanent_uuid: "76ffb1129a0641fd91d08eb80e96c5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44429 } } peers { permanent_uuid: "2f37d9f42ab24153831e7a1078b56049" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43205 } }
I20260926 07:25:57.027205 11149 raft_consensus.cc:348] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe422d7074432788315ca5ecce415f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39639 } } peers { permanent_uuid: "76ffb1129a0641fd91d08eb80e96c5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44429 } } peers { permanent_uuid: "2f37d9f42ab24153831e7a1078b56049" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43205 } }
I20260926 07:25:57.027426 11149 raft_consensus.cc:374] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:57.027484 11149 raft_consensus.cc:729] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efe422d7074432788315ca5ecce415f, State: Initialized, Role: FOLLOWER
I20260926 07:25:57.027527 11131 heartbeater.cc:503] Master 127.0.98.62:33235 was elected leader, sending a full tablet report...
I20260926 07:25:57.027875 11148 ts_tablet_manager.cc:1428] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:25:57.027886 11149 consensus_queue.cc:260] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe422d7074432788315ca5ecce415f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39639 } } peers { permanent_uuid: "76ffb1129a0641fd91d08eb80e96c5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44429 } } peers { permanent_uuid: "2f37d9f42ab24153831e7a1078b56049" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43205 } }
I20260926 07:25:57.028795 11149 ts_tablet_manager.cc:1428] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 07:25:57.029134 10863 heartbeater.cc:503] Master 127.0.98.62:33235 was elected leader, sending a full tablet report...
I20260926 07:25:57.029294 11150 tablet_bootstrap.cc:492] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac: No bootstrap required, opened a new log
I20260926 07:25:57.029446 11150 ts_tablet_manager.cc:1397] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260926 07:25:57.032876 11150 raft_consensus.cc:348] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe422d7074432788315ca5ecce415f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39639 } } peers { permanent_uuid: "76ffb1129a0641fd91d08eb80e96c5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44429 } } peers { permanent_uuid: "2f37d9f42ab24153831e7a1078b56049" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43205 } }
I20260926 07:25:57.033069 11150 raft_consensus.cc:374] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:25:57.033103 11150 raft_consensus.cc:729] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ffb1129a0641fd91d08eb80e96c5ac, State: Initialized, Role: FOLLOWER
I20260926 07:25:57.033453 11150 consensus_queue.cc:260] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe422d7074432788315ca5ecce415f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39639 } } peers { permanent_uuid: "76ffb1129a0641fd91d08eb80e96c5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44429 } } peers { permanent_uuid: "2f37d9f42ab24153831e7a1078b56049" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43205 } }
I20260926 07:25:57.033995 10997 heartbeater.cc:503] Master 127.0.98.62:33235 was elected leader, sending a full tablet report...
I20260926 07:25:57.034174 11150 ts_tablet_manager.cc:1428] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260926 07:25:57.206533 11132 tablet.cc:2369] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:25:57.230755 10998 tablet.cc:2369] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:25:57.250146 11155 raft_consensus.cc:482] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:25:57.250415 11155 raft_consensus.cc:504] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe422d7074432788315ca5ecce415f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39639 } } peers { permanent_uuid: "76ffb1129a0641fd91d08eb80e96c5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44429 } } peers { permanent_uuid: "2f37d9f42ab24153831e7a1078b56049" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43205 } }
I20260926 07:25:57.251652 11155 leader_election.cc:288] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76ffb1129a0641fd91d08eb80e96c5ac (127.0.98.2:44429), 2f37d9f42ab24153831e7a1078b56049 (127.0.98.3:43205)
I20260926 07:25:57.256982 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:57.252080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51577 (local address 127.0.98.2:44429)
0926 07:25:57.252216 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:57.252232 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:57.252503 (+ 271us) server_negotiation.cc:407] Connection header received
0926 07:25:57.252680 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:57.252684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:57.252734 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:57.252830 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:57.253123 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.253924 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.255721 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.256328 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.256440 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:57.256506 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:57.256779 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 07:25:57.256848 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:25:57.256982 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:57.252098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51551 (local address 127.0.98.3:43205)
0926 07:25:57.252232 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:57.252236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:57.252618 (+ 382us) server_negotiation.cc:407] Connection header received
0926 07:25:57.252778 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:57.252781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:57.252833 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:57.252918 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:25:57.253240 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.253977 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.255729 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.256328 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.256440 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:57.256506 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:57.256779 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 07:25:57.256848 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 07:25:57.257062 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:57.251982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43205 (local address 127.0.98.1:51551)
0926 07:25:57.252512 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:57.252524 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:57.252655 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:57.252937 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:57.252940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:57.252955 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:57.253066 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:57.253070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.254080 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.254083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:57.255589 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:57.255598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.256472 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.256476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:57.256662 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:57.256678 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 07:25:57.256743 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 07:25:57.256866 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":166,"thread_start_us":98,"threads_started":1}
I20260926 07:25:57.257063 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:57.251982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44429 (local address 127.0.98.1:51577)
0926 07:25:57.252395 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:57.252407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:25:57.252563 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:25:57.252840 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:25:57.252842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:25:57.252857 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:25:57.252997 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:57.253002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.254067 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.254072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:57.255587 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:25:57.255598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.256472 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.256476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:25:57.256626 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:57.256661 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:25:57.256743 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 07:25:57.256866 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":161,"thread_start_us":138,"threads_started":1}
I20260926 07:25:57.257835 10951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862ac19de76e495bbc62c3f0a57be726" candidate_uuid: "1efe422d7074432788315ca5ecce415f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ffb1129a0641fd91d08eb80e96c5ac" is_pre_election: true
I20260926 07:25:57.257843 11085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862ac19de76e495bbc62c3f0a57be726" candidate_uuid: "1efe422d7074432788315ca5ecce415f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f37d9f42ab24153831e7a1078b56049" is_pre_election: true
I20260926 07:25:57.258144 11085 raft_consensus.cc:2454] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1efe422d7074432788315ca5ecce415f in term 0.
I20260926 07:25:57.258147 10951 raft_consensus.cc:2454] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1efe422d7074432788315ca5ecce415f in term 0.
I20260926 07:25:57.258622 10753 leader_election.cc:302] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe422d7074432788315ca5ecce415f, 2f37d9f42ab24153831e7a1078b56049; no voters:
I20260926 07:25:57.258917 11155 raft_consensus.cc:2781] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:25:57.258999 11155 raft_consensus.cc:482] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:25:57.259066 11155 raft_consensus.cc:3037] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [term 0 FOLLOWER]: Advancing to term 1
W20260926 07:25:57.259261 10864 tablet.cc:2369] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:25:57.260257 11155 raft_consensus.cc:504] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe422d7074432788315ca5ecce415f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39639 } } peers { permanent_uuid: "76ffb1129a0641fd91d08eb80e96c5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44429 } } peers { permanent_uuid: "2f37d9f42ab24153831e7a1078b56049" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43205 } }
I20260926 07:25:57.260645 11155 leader_election.cc:288] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [CANDIDATE]: Term 1 election: Requested vote from peers 76ffb1129a0641fd91d08eb80e96c5ac (127.0.98.2:44429), 2f37d9f42ab24153831e7a1078b56049 (127.0.98.3:43205)
I20260926 07:25:57.260998 10951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862ac19de76e495bbc62c3f0a57be726" candidate_uuid: "1efe422d7074432788315ca5ecce415f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ffb1129a0641fd91d08eb80e96c5ac"
I20260926 07:25:57.260998 11085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "862ac19de76e495bbc62c3f0a57be726" candidate_uuid: "1efe422d7074432788315ca5ecce415f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f37d9f42ab24153831e7a1078b56049"
I20260926 07:25:57.261123 10951 raft_consensus.cc:3037] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:57.261123 11085 raft_consensus.cc:3037] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:25:57.262374 11085 raft_consensus.cc:2454] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe422d7074432788315ca5ecce415f in term 1.
I20260926 07:25:57.262374 10951 raft_consensus.cc:2454] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1efe422d7074432788315ca5ecce415f in term 1.
I20260926 07:25:57.262667 10750 leader_election.cc:302] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efe422d7074432788315ca5ecce415f, 76ffb1129a0641fd91d08eb80e96c5ac; no voters:
I20260926 07:25:57.262986 11155 raft_consensus.cc:2781] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:25:57.263208 11155 raft_consensus.cc:686] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [term 1 LEADER]: Becoming Leader. State: Replica: 1efe422d7074432788315ca5ecce415f, State: Running, Role: LEADER
I20260926 07:25:57.263531 11155 consensus_queue.cc:237] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe422d7074432788315ca5ecce415f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39639 } } peers { permanent_uuid: "76ffb1129a0641fd91d08eb80e96c5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44429 } } peers { permanent_uuid: "2f37d9f42ab24153831e7a1078b56049" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43205 } }
I20260926 07:25:57.266973 10667 catalog_manager.cc:5495] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1efe422d7074432788315ca5ecce415f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1efe422d7074432788315ca5ecce415f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efe422d7074432788315ca5ecce415f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76ffb1129a0641fd91d08eb80e96c5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f37d9f42ab24153831e7a1078b56049" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43205 } health_report { overall_health: UNKNOWN } } }
I20260926 07:25:57.298970 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:57.294825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.0.98.1:39639)
0926 07:25:57.294997 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:25:57.295000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:25:57.295019 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:25:57.295201 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:25:57.295204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:25:57.295288 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:25:57.295434 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:25:57.295780 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.296364 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.297331 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:25:57.297624 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:25:57.297758 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:25:57.298102 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:25:57.298543 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:25:57.298706 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 07:25:57.298745 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 07:25:57.298805 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 07:25:57.304720 10951 raft_consensus.cc:1261] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac [term 1 FOLLOWER]: Refusing update from remote peer 1efe422d7074432788315ca5ecce415f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:57.304739 11085 raft_consensus.cc:1261] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049 [term 1 FOLLOWER]: Refusing update from remote peer 1efe422d7074432788315ca5ecce415f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:25:57.305495 11159 consensus_queue.cc:1035] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f37d9f42ab24153831e7a1078b56049" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:57.305691 11155 consensus_queue.cc:1035] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ffb1129a0641fd91d08eb80e96c5ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:25:57.312299 11162 mvcc.cc:205] Tried to move back new op lower bound from 7333509354713276416 to 7333509354554028032. Current Snapshot: MvccSnapshot[applied={T|T < 7333509354713276416}]
I20260926 07:25:57.313607 11164 mvcc.cc:205] Tried to move back new op lower bound from 7333509354713276416 to 7333509354554028032. Current Snapshot: MvccSnapshot[applied={T|T < 7333509354713276416}]
I20260926 07:25:57.314704 11163 mvcc.cc:205] Tried to move back new op lower bound from 7333509354713276416 to 7333509354554028032. Current Snapshot: MvccSnapshot[applied={T|T < 7333509354713276416}]
I20260926 07:25:57.321482 10667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:57.321626 10667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:25:57.323979 10667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3467fc7cb38461790bdb1d9a7c7a925: Sending DeleteTablet for 3 replicas of tablet 862ac19de76e495bbc62c3f0a57be726
I20260926 07:25:57.324975 10797 tablet_service.cc:1514] Processing DeleteTablet for tablet 862ac19de76e495bbc62c3f0a57be726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55516
I20260926 07:25:57.324990 11065 tablet_service.cc:1514] Processing DeleteTablet for tablet 862ac19de76e495bbc62c3f0a57be726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49988
I20260926 07:25:57.325069 10931 tablet_service.cc:1514] Processing DeleteTablet for tablet 862ac19de76e495bbc62c3f0a57be726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:25:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34902
I20260926 07:25:57.325587 11172 tablet_replica.cc:330] T 862ac19de76e495bbc62c3f0a57be726 P 1efe422d7074432788315ca5ecce415f: stopping tablet replica
I20260926 07:25:57.325665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 10734
I20260926 07:25:57.325814 11174 tablet_replica.cc:330] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac: stopping tablet replica
I20260926 07:25:57.326838 11174 raft_consensus.cc:2229] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:57.327178 11173 tablet_replica.cc:330] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049: stopping tablet replica
I20260926 07:25:57.327198 11174 raft_consensus.cc:2258] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:57.327471 11173 raft_consensus.cc:2229] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:25:57.327744 11173 raft_consensus.cc:2258] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:25:57.328631 11174 ts_tablet_manager.cc:1905] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:57.329831 11173 ts_tablet_manager.cc:1905] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:25:57.330904 11174 ts_tablet_manager.cc:1918] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:57.331017 11174 log.cc:1196] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/862ac19de76e495bbc62c3f0a57be726
I20260926 07:25:57.331326 11174 ts_tablet_manager.cc:1939] T 862ac19de76e495bbc62c3f0a57be726 P 76ffb1129a0641fd91d08eb80e96c5ac: Deleting consensus metadata
I20260926 07:25:57.332402 10656 catalog_manager.cc:4841] TS 76ffb1129a0641fd91d08eb80e96c5ac (127.0.98.2:44429): tablet 862ac19de76e495bbc62c3f0a57be726 (table test-table [id=9f16adbdaf884372ba4756592ac3c7d3]) successfully deleted
I20260926 07:25:57.333160 11173 ts_tablet_manager.cc:1918] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:25:57.333266 11173 log.cc:1196] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/862ac19de76e495bbc62c3f0a57be726
I20260926 07:25:57.333599 11173 ts_tablet_manager.cc:1939] T 862ac19de76e495bbc62c3f0a57be726 P 2f37d9f42ab24153831e7a1078b56049: Deleting consensus metadata
W20260926 07:25:57.334129 10656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39639 (error 108)
I20260926 07:25:57.334348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 10868
I20260926 07:25:57.335762 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:25:57.335424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39639 (local address 127.0.0.1:55532)
0926 07:25:57.335577 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 07:25:57.335659 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:25:57.335938 10656 catalog_manager.cc:4584] TS 1efe422d7074432788315ca5ecce415f (127.0.98.1:39639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 862ac19de76e495bbc62c3f0a57be726: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39639: connect: Connection refused (error 111)
I20260926 07:25:57.337288 10659 catalog_manager.cc:4841] TS 2f37d9f42ab24153831e7a1078b56049 (127.0.98.3:43205): tablet 862ac19de76e495bbc62c3f0a57be726 (table test-table [id=9f16adbdaf884372ba4756592ac3c7d3]) successfully deleted
I20260926 07:25:57.341451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 11002
I20260926 07:25:57.347308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 10640
2026-09-26T07:25:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2429 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:25:57 dist-test-slave-g04g krb5kdc[11178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:25:57 dist-test-slave-g04g krb5kdc[11178](info): set up 2 sockets
Sep 26 07:25:57 dist-test-slave-g04g krb5kdc[11178](info): commencing operation
krb5kdc: starting...
W20260926 07:25:58.440066 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:25:58 dist-test-slave-g04g krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:25:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:25:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 07:25:58.605564 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_EJaZL/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/postgres -l logfile start
2026-09-26 07:26:01.221 UTC [11208] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 07:26:01.222 UTC [11208] LOG: listening on IPv4 address "127.0.98.20", port 57841
2026-09-26 07:26:01.225 UTC [11208] LOG: listening on Unix socket "/tmp/.s.PGSQL.57841"
2026-09-26 07:26:01.259 UTC [11211] LOG: database system was shut down at 2026-09-26 07:25:59 UTC
2026-09-26 07:26:01.263 UTC [11208] LOG: database system is ready to accept connections
I20260926 07:26:02.295804 392 mini_postgres.cc:96] Postgres bound to 57841
2026-09-26 07:26:02.302 UTC [11219] FATAL: database "slave" does not exist
127.0.98.20:57841 - accepting connections
I20260926 07:26:02.302628 392 mini_ranger.cc:161] Starting Ranger...
I20260926 07:26:02.317353 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 07:26:02.832574 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 07:26:02.832698 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-admin
2026-09-26 07:26:03,105 [I] DB FLAVOR :POSTGRES
2026-09-26 07:26:03,106 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:26:03,106 [I] Checking connection..
2026-09-26 07:26:03,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 07:26:03,363 [I] Checking connection passed.
2026-09-26 07:26:03,363 [I] --------- Verifying version history table ---------
2026-09-26 07:26:03,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:26:03,597 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:26:03,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:26:03,835 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:26:03,836 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 07:26:03,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 07:26:04.086 UTC [11283] WARNING: there is no transaction in progress
2026-09-26 07:26:04,092 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 07:26:04,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:26:04,348 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 07:26:04,348 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:26:04,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 07:26:04,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 07:26:04,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g04g.c.gcp-upstream.internal','N') ;"
2026-09-26 07:26:05,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:26:05,336 [I] Table x_portal_user does not exist in database ranger
2026-09-26 07:26:05,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:26:05,585 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 07:26:05,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:26:05,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:26:06,078 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 07:26:06,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 07:26:07.110 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.128 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.144 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.160 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.177 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.195 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.257 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.264 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.278 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.289 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.299 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.313 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.323 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.330 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.336 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:07.343 UTC [11391] WARNING: there is no transaction in progress
2026-09-26 07:26:08,015 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 07:26:08,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 07:26:08,264 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 07:26:08,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:26:08,491 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 07:26:08,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:26:08,744 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 07:26:08,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:26:09,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:26:09,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g04g.c.gcp-upstream.internal';"
2026-09-26 07:26:09,503 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 07:26:09,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 07:26:09,760 [I] DB_PATCHES have already been applied
I20260926 07:26:09.765978 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_EJaZL/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 07:26:09.766070 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 07:26:09.769485 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:50465
Sep 26, 2026 7:26:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:26:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:26:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:26:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:26:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:26:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:26:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 07:26:10.886721 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.584s user 0.005s sys 0.005s
Sep 26, 2026 7:26:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:26:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 7:26:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:26:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:26:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:26:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:26:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:26:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:26:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:26:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:26:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:26:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:26:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:26:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:26:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:26:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:26:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:26:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 07:26:32.935379 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 07:26:33 dist-test-slave-g04g krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407593, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 07:26:33.024618 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 07:26:33.034116 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 07:26:33.970501 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 07:26:33.970584 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-kms
I20260926 07:26:33.970681 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:57841
2026-09-26 07:26:34,165 [I] DB FLAVOR :POSTGRES
2026-09-26 07:26:34,165 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:26:34,165 [I] Checking connection
2026-09-26 07:26:34,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 07:26:34,413 [I] connection success
2026-09-26 07:26:34,414 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 07:26:34,414 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 07:26:34,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 07:26:34,649 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 07:26:34,649 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:26:34,650 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 07:26:34,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 07:26:34,923 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 07:26:35.388288 392 mini_ranger_kms.cc:310] Created kms service
I20260926 07:26:35.554158 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 07:26:35.597705 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 07:26:35 dist-test-slave-g04g krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407593, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:26:36.004568 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 07:26:36.004747 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 07:26:36.004777 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 07:26:36.008940 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11571
I20260926 07:26:36.009083 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:37665
Sep 26, 2026 7:26:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:26:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:26:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:26:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:26:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:26:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 07:26:38.277865 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.253s user 0.006s sys 0.172s
I20260926 07:26:38.278012 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 07:26:38.278141 392 mini_ranger_kms.cc:401] 127.0.98.20:37665/kms/v1/keys
Sep 26 07:26:38 dist-test-slave-g04g krb5kdc[11178](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407598, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 07:26:38 dist-test-slave-g04g krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407598, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 07:26:39 dist-test-slave-g04g krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407598, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 07:26:41 dist-test-slave-g04g krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:26:41.478881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33741
--ranger_config_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37665/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:26:41.590746 11622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:26:41.591092 11622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:26:41.591176 11622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:26:41.595346 11622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:26:41.595402 11622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:26:41.595422 11622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:26:41.595464 11622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:26:41.595492 11622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:26:41.600826 11622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:37665/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33741
--ranger_config_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:26:41.601847 11622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:26:41.602854 11622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:26:41.611244 11630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:26:41.611232 11627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:26:41.611235 11628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:26:41.611778 11622 server_base.cc:1034] running on GCE node
I20260926 07:26:41.612341 11622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:26:41.613387 11622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:26:41.614563 11622 hybrid_clock.cc:648] HybridClock initialized: now 1790407601614542 us; error 37 us; skew 500 ppm
Sep 26 07:26:41 dist-test-slave-g04g krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:26:41.617507 11622 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:26:41.618454 11622 webserver.cc:460] Webserver started at http://127.0.98.62:43991/ using document root <none> and password file <none>
I20260926 07:26:41.619019 11622 fs_manager.cc:359] Metadata directory not provided
I20260926 07:26:41.619136 11622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:26:41.619359 11622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:26:41 dist-test-slave-g04g krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:26:41.666276 11622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "0613bf6ec5fb497dbc11373ed1d216c2"
format_stamp: "Formatted at 2026-09-26 07:26:41 on dist-test-slave-g04g"
server_key: "3448187a206f744ccc7dfb6cbd69a544"
server_key_iv: "1c61ffe4a6422411d1d122aafe4e0304"
server_key_version: "kuduclusterkey@0"
I20260926 07:26:41.666914 11622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "0613bf6ec5fb497dbc11373ed1d216c2"
format_stamp: "Formatted at 2026-09-26 07:26:41 on dist-test-slave-g04g"
server_key: "3448187a206f744ccc7dfb6cbd69a544"
server_key_iv: "1c61ffe4a6422411d1d122aafe4e0304"
server_key_version: "kuduclusterkey@0"
I20260926 07:26:41.670760 11622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260926 07:26:41.687671 11637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:26:41.689123 11622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:26:41.689262 11622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "0613bf6ec5fb497dbc11373ed1d216c2"
format_stamp: "Formatted at 2026-09-26 07:26:41 on dist-test-slave-g04g"
server_key: "3448187a206f744ccc7dfb6cbd69a544"
server_key_iv: "1c61ffe4a6422411d1d122aafe4e0304"
server_key_version: "kuduclusterkey@0"
I20260926 07:26:41.689421 11622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:26:41.730424 11622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:26:41.733168 11622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:26:41.733330 11622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:26:41.740489 11622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33741
I20260926 07:26:41.740500 11689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33741 every 8 connection(s)
I20260926 07:26:41.741407 11622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:26:41.744031 11690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:26:41.746033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 11622
I20260926 07:26:41.746156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:26:41.746461 392 external_mini_cluster.cc:1428] Setting key 1e6232500a455e66e657d14697438f6e
I20260926 07:26:41.750412 11690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2: Bootstrap starting.
I20260926 07:26:41.752528 11690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2: Neither blocks nor log segments found. Creating new log.
I20260926 07:26:41.753250 11690 log.cc:824] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:26:41 dist-test-slave-g04g krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:26:41.755295 11690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2: No bootstrap required, opened a new log
I20260926 07:26:41.758690 11690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0613bf6ec5fb497dbc11373ed1d216c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } }
I20260926 07:26:41.758917 11690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:26:41.758994 11690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0613bf6ec5fb497dbc11373ed1d216c2, State: Initialized, Role: FOLLOWER
I20260926 07:26:41.759465 11690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0613bf6ec5fb497dbc11373ed1d216c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } }
I20260926 07:26:41.759609 11690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:26:41.759667 11690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:26:41.759770 11690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:26:41.760540 11690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0613bf6ec5fb497dbc11373ed1d216c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } }
I20260926 07:26:41.760586 11693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:41.747854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59228 (local address 127.0.98.62:33741)
0926 07:26:41.748283 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 07:26:41.748297 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:26:41.748335 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:26:41.748975 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:26:41.748992 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:26:41.749293 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:26:41.749609 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:26:41.750339 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:41.751391 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:41.752214 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:41.752714 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:41.752873 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:41.755626 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:26:41.755632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:26:41.755644 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:26:41.755668 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:26:41.757787 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:26:41.758214 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:26:41.758219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:26:41.758225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:26:41.758355 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:26:41.758625 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:26:41.758628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:26:41.758630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:26:41.759020 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:26:41.759236 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 07:26:41.759516 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 07:26:41.759933 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":128,"threads_started":1}
I20260926 07:26:41.760921 11690 leader_election.cc:302] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0613bf6ec5fb497dbc11373ed1d216c2; no voters:
I20260926 07:26:41.761247 11690 leader_election.cc:288] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:26:41.761413 11695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:26:41.761627 11695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [term 1 LEADER]: Becoming Leader. State: Replica: 0613bf6ec5fb497dbc11373ed1d216c2, State: Running, Role: LEADER
I20260926 07:26:41.761905 11695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0613bf6ec5fb497dbc11373ed1d216c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } }
I20260926 07:26:41.762393 11690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:26:41.763686 11696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0613bf6ec5fb497dbc11373ed1d216c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0613bf6ec5fb497dbc11373ed1d216c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } } }
I20260926 07:26:41.763830 11696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:26:41.763690 11697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0613bf6ec5fb497dbc11373ed1d216c2. Latest consensus state: current_term: 1 leader_uuid: "0613bf6ec5fb497dbc11373ed1d216c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0613bf6ec5fb497dbc11373ed1d216c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33741 } } }
I20260926 07:26:41.763923 11697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:26:41.764577 11690 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:26:41.764843 11706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:26:41.769982 11706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:26:41.775053 11706 catalog_manager.cc:1347] Generated new cluster ID: 2caa91d439614e969c4a326c8b201bc2
I20260926 07:26:41.775108 11706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:26:41.796398 11706 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:26:41.797341 11706 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:26:41.801707 11706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2: Generated new TSK 0
I20260926 07:26:41.802387 11706 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:26:42.494167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33741
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:26:42.640043 11735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:26:42.640319 11735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:26:42.640410 11735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:26:42.644727 11735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:26:42.644812 11735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:26:42.644899 11735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:26:42.651252 11735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:37665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:26:42.652321 11735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:26:42.653411 11735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:26:42.663303 11742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:26:42.663744 11745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:26:42.663288 11743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:26:42.664310 11735 server_base.cc:1034] running on GCE node
I20260926 07:26:42.664728 11735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:26:42.665407 11735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:26:42.666599 11735 hybrid_clock.cc:648] HybridClock initialized: now 1790407602666567 us; error 53 us; skew 500 ppm
Sep 26 07:26:42 dist-test-slave-g04g krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:26:42.669943 11735 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:26:42.671061 11735 webserver.cc:460] Webserver started at http://127.0.98.1:36825/ using document root <none> and password file <none>
I20260926 07:26:42.671774 11735 fs_manager.cc:359] Metadata directory not provided
I20260926 07:26:42.671864 11735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:26:42.672098 11735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:26:42 dist-test-slave-g04g krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:26:42.688103 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "3fc5db53ecf34ca6926b4f1852055ee0"
format_stamp: "Formatted at 2026-09-26 07:26:42 on dist-test-slave-g04g"
server_key: "6cbe3c60de058e75f05b310dfac2f732"
server_key_iv: "d57e50fc35dcae071b37807f5356495e"
server_key_version: "kuduclusterkey@0"
I20260926 07:26:42.690169 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3fc5db53ecf34ca6926b4f1852055ee0"
format_stamp: "Formatted at 2026-09-26 07:26:42 on dist-test-slave-g04g"
server_key: "6cbe3c60de058e75f05b310dfac2f732"
server_key_iv: "d57e50fc35dcae071b37807f5356495e"
server_key_version: "kuduclusterkey@0"
I20260926 07:26:42.697067 11735 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 07:26:42.734370 11752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:26:42.735991 11735 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.003s sys 0.000s
I20260926 07:26:42.736147 11735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "3fc5db53ecf34ca6926b4f1852055ee0"
format_stamp: "Formatted at 2026-09-26 07:26:42 on dist-test-slave-g04g"
server_key: "6cbe3c60de058e75f05b310dfac2f732"
server_key_iv: "d57e50fc35dcae071b37807f5356495e"
server_key_version: "kuduclusterkey@0"
I20260926 07:26:42.736291 11735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:26:42.751149 11735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:26:42.754654 11735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:26:42.754894 11735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:26:42.755648 11735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:26:42.756692 11735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:26:42.756799 11735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:26:42.756922 11735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:26:42.757002 11735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:26:42.766767 11735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44293
I20260926 07:26:42.767956 11735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:26:42.770069 11867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44293 every 8 connection(s)
I20260926 07:26:42.774890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 11735
I20260926 07:26:42.775023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:26:42.775399 392 external_mini_cluster.cc:1428] Setting key 4694164af42fa45fda711b27d0e8dd18
Sep 26 07:26:42 dist-test-slave-g04g krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 07:26:42.790024 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:42.772738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45025 (local address 127.0.98.62:33741)
0926 07:26:42.773002 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 07:26:42.773011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:26:42.773437 (+ 426us) server_negotiation.cc:407] Connection header received
0926 07:26:42.774676 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:26:42.774680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:26:42.774738 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:26:42.774829 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:26:42.779288 (+ 4459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:42.780143 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:42.781878 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:42.782239 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:42.782330 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:42.785464 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:26:42.785468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:26:42.785471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:26:42.785498 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:26:42.787403 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:26:42.788040 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:26:42.788044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:26:42.788046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:26:42.788142 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:26:42.788594 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:26:42.788598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:26:42.788600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:26:42.788841 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:26:42.788919 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:26:42.789701 (+ 782us) server_negotiation.cc:299] Negotiation successful
0926 07:26:42.789839 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260926 07:26:42.791785 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:42.772747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33741 (local address 127.0.98.1:45025)
0926 07:26:42.773275 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 07:26:42.773332 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 07:26:42.774428 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:26:42.776095 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:26:42.776105 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:26:42.776453 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:26:42.779016 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:42.779108 (+ 92us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:42.780280 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:42.780284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:42.781725 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:42.781736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:42.782552 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:42.782556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:42.782762 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:42.783525 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:26:42.783548 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:26:42.785266 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:26:42.787533 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:26:42.787539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:26:42.787556 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:26:42.787915 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:26:42.788277 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:26:42.788280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:26:42.788282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:26:42.788474 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:26:42.789010 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:26:42.789017 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:26:42.789397 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 07:26:42.790214 (+ 817us) client_negotiation.cc:240] Negotiation successful
0926 07:26:42.790589 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 07:26:42.793525 11868 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33741
I20260926 07:26:42.793797 11868 heartbeater.cc:464] Registering TS with master...
I20260926 07:26:42.794409 11868 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
I20260926 07:26:42.796166 11654 ts_manager.cc:194] Registered new tserver with Master: 3fc5db53ecf34ca6926b4f1852055ee0 (127.0.98.1:44293)
I20260926 07:26:42.797705 11654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:26:42.814896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33741
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:26:42 dist-test-slave-g04g krb5kdc[11178](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 07:26:42.980020 11874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:26:42.980469 11874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:26:42.980604 11874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:26:42.987159 11874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:26:42.987385 11874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:26:42.987555 11874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:26:42.995637 11874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:37665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:26:42.996978 11874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:26:42.998327 11874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:26:43.011082 11883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:26:43.011209 11880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:26:43.011098 11881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:26:43.012481 11874 server_base.cc:1034] running on GCE node
I20260926 07:26:43.012948 11874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:26:43.013746 11874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:26:43.014940 11874 hybrid_clock.cc:648] HybridClock initialized: now 1790407603014912 us; error 42 us; skew 500 ppm
Sep 26 07:26:43 dist-test-slave-g04g krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:26:43.018628 11874 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:26:43.019776 11874 webserver.cc:460] Webserver started at http://127.0.98.2:34125/ using document root <none> and password file <none>
I20260926 07:26:43.020473 11874 fs_manager.cc:359] Metadata directory not provided
I20260926 07:26:43.020560 11874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:26:43.020792 11874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:26:43 dist-test-slave-g04g krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:26:43.035490 11874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "84500f2208004152a192bedcfd5a8342"
format_stamp: "Formatted at 2026-09-26 07:26:43 on dist-test-slave-g04g"
server_key: "883f6801f9a39ff0495e43ea56866b55"
server_key_iv: "2c6cdbc164f040569bdec9a3805750c5"
server_key_version: "kuduclusterkey@0"
I20260926 07:26:43.036304 11874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84500f2208004152a192bedcfd5a8342"
format_stamp: "Formatted at 2026-09-26 07:26:43 on dist-test-slave-g04g"
server_key: "883f6801f9a39ff0495e43ea56866b55"
server_key_iv: "2c6cdbc164f040569bdec9a3805750c5"
server_key_version: "kuduclusterkey@0"
I20260926 07:26:43.047530 11874 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260926 07:26:43.071182 11890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:26:43.073938 11874 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260926 07:26:43.074134 11874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "84500f2208004152a192bedcfd5a8342"
format_stamp: "Formatted at 2026-09-26 07:26:43 on dist-test-slave-g04g"
server_key: "883f6801f9a39ff0495e43ea56866b55"
server_key_iv: "2c6cdbc164f040569bdec9a3805750c5"
server_key_version: "kuduclusterkey@0"
I20260926 07:26:43.074301 11874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:26:43.088852 11874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:26:43.092294 11874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:26:43.092554 11874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:26:43.093261 11874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:26:43.094321 11874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:26:43.094384 11874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:26:43.094439 11874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:26:43.094471 11874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:26:43.106302 11874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33399
I20260926 07:26:43.106433 12003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33399 every 8 connection(s)
I20260926 07:26:43.107513 11874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:26:43.109666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 11874
I20260926 07:26:43.109798 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:26:43.110109 392 external_mini_cluster.cc:1428] Setting key a215422bd389b5da637469c07cac417f
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 26 07:26:43 dist-test-slave-g04g krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 07:26:43.143863 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.111215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42379 (local address 127.0.98.62:33741)
0926 07:26:43.111350 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:26:43.111354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:26:43.115643 (+ 4289us) server_negotiation.cc:407] Connection header received
0926 07:26:43.117437 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:26:43.117442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:26:43.117498 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:26:43.117588 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:26:43.119158 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.119986 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.121776 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.122187 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.122313 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.138719 (+ 16406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:26:43.138724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:26:43.138727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:26:43.138772 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:26:43.140670 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:26:43.141443 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:26:43.141447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:26:43.141448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:26:43.141539 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:26:43.142433 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:26:43.142437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:26:43.142439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:26:43.142672 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:26:43.142765 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:26:43.143533 (+ 768us) server_negotiation.cc:299] Negotiation successful
0926 07:26:43.143668 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:26:43.145779 12006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.110326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33741 (local address 127.0.98.2:42379)
0926 07:26:43.115440 (+ 5114us) negotiation.cc:107] Waiting for socket to connect
0926 07:26:43.115509 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 07:26:43.116772 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:26:43.117909 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:26:43.117919 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:26:43.118274 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:26:43.118968 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.118988 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.120159 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.120163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.121574 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.121584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.122515 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.122521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.122764 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.123502 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:26:43.123522 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:26:43.134051 (+ 10529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:26:43.140846 (+ 6795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:26:43.140867 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:26:43.140890 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:26:43.141315 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:26:43.142073 (+ 758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:26:43.142076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:26:43.142079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:26:43.142311 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:26:43.142820 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:26:43.142827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:26:43.143229 (+ 402us) client_negotiation.cc:769] Sending connection context
0926 07:26:43.144346 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0926 07:26:43.144712 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4850,"thread_start_us":136,"threads_started":1}
I20260926 07:26:43.147365 12004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33741
I20260926 07:26:43.147610 12004 heartbeater.cc:464] Registering TS with master...
I20260926 07:26:43.148165 12004 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
I20260926 07:26:43.149307 11654 ts_manager.cc:194] Registered new tserver with Master: 84500f2208004152a192bedcfd5a8342 (127.0.98.2:33399)
I20260926 07:26:43.150018 11654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:26:43.166697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33741
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:26:43.328569 12009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:26:43.328897 12009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:26:43.328958 12009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 26 07:26:43 dist-test-slave-g04g krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 07:26:43.335281 12009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:26:43.335350 12009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:26:43.335433 12009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:26:43.342967 12009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:37665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:26:43.344285 12009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:26:43.345654 12009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:26:43.358918 12015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:26:43.359252 12014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:26:43.362138 12009 server_base.cc:1034] running on GCE node
W20260926 07:26:43.362421 12017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:26:43.363085 12009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:26:43.363890 12009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:26:43.365054 12009 hybrid_clock.cc:648] HybridClock initialized: now 1790407603364990 us; error 82 us; skew 500 ppm
Sep 26 07:26:43 dist-test-slave-g04g krb5kdc[11178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:26:43.368492 12009 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:26:43.369612 12009 webserver.cc:460] Webserver started at http://127.0.98.3:42541/ using document root <none> and password file <none>
I20260926 07:26:43.370330 12009 fs_manager.cc:359] Metadata directory not provided
I20260926 07:26:43.370424 12009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:26:43.370666 12009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:26:43 dist-test-slave-g04g krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:26:43.386605 12009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "8dd209a8ad834b4099896c4ecc7b7fd2"
format_stamp: "Formatted at 2026-09-26 07:26:43 on dist-test-slave-g04g"
server_key: "4cbadcd487e1e060ba9c06c79ded87d1"
server_key_iv: "579d7e908e5ce9e2b0e30416d34864dd"
server_key_version: "kuduclusterkey@0"
I20260926 07:26:43.393553 12009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8dd209a8ad834b4099896c4ecc7b7fd2"
format_stamp: "Formatted at 2026-09-26 07:26:43 on dist-test-slave-g04g"
server_key: "4cbadcd487e1e060ba9c06c79ded87d1"
server_key_iv: "579d7e908e5ce9e2b0e30416d34864dd"
server_key_version: "kuduclusterkey@0"
I20260926 07:26:43.405364 12009 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.000s sys 0.004s
I20260926 07:26:43.431473 12024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:26:43.433223 12009 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260926 07:26:43.433372 12009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "8dd209a8ad834b4099896c4ecc7b7fd2"
format_stamp: "Formatted at 2026-09-26 07:26:43 on dist-test-slave-g04g"
server_key: "4cbadcd487e1e060ba9c06c79ded87d1"
server_key_iv: "579d7e908e5ce9e2b0e30416d34864dd"
server_key_version: "kuduclusterkey@0"
I20260926 07:26:43.433530 12009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:26:43.459389 12009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:26:43.463614 12009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:26:43.463886 12009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:26:43.464723 12009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:26:43.465852 12009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:26:43.465981 12009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:26:43.466131 12009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:26:43.466289 12009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:26:43.481920 12009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42381
I20260926 07:26:43.483245 12009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:26:43.485822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 12009
I20260926 07:26:43.485961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:26:43.486285 392 external_mini_cluster.cc:1428] Setting key 6690f6feadcbca4a90b62cedb7c7adfb
I20260926 07:26:43.492619 12137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42381 every 8 connection(s)
Sep 26 07:26:43 dist-test-slave-g04g krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:26:43.510443 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.494805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34041 (local address 127.0.98.62:33741)
0926 07:26:43.494943 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:26:43.494948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:26:43.496711 (+ 1763us) server_negotiation.cc:407] Connection header received
0926 07:26:43.497952 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:26:43.497955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:26:43.498008 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:26:43.498061 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:26:43.499754 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.500524 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.502188 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.502524 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.502592 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.506302 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:26:43.506306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:26:43.506309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:26:43.506337 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:26:43.508068 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:26:43.508676 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:26:43.508680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:26:43.508682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:26:43.508768 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:26:43.509177 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:26:43.509181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:26:43.509182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:26:43.509392 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:26:43.509468 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:26:43.510156 (+ 688us) server_negotiation.cc:299] Negotiation successful
0926 07:26:43.510280 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:26:43.512403 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.495372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33741 (local address 127.0.98.3:34041)
0926 07:26:43.496563 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0926 07:26:43.496619 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:26:43.497708 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:26:43.498489 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:26:43.498498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:26:43.498863 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:26:43.499597 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.499616 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.500671 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.500674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.502040 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.502050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.502806 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.502810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.502998 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.503913 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:26:43.503933 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:26:43.506098 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:26:43.508210 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:26:43.508216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:26:43.508232 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:26:43.508559 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:26:43.508877 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:26:43.508880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:26:43.508882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:26:43.509055 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:26:43.509531 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:26:43.509537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:26:43.509872 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 07:26:43.510864 (+ 992us) client_negotiation.cc:240] Negotiation successful
0926 07:26:43.511275 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":129,"threads_started":1}
I20260926 07:26:43.513912 12138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33741
I20260926 07:26:43.514151 12138 heartbeater.cc:464] Registering TS with master...
I20260926 07:26:43.514701 12138 heartbeater.cc:511] Master 127.0.98.62:33741 requested a full tablet report, sending...
I20260926 07:26:43.515812 11654 ts_manager.cc:194] Registered new tserver with Master: 8dd209a8ad834b4099896c4ecc7b7fd2 (127.0.98.3:42381)
I20260926 07:26:43.516549 11654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34041
I20260926 07:26:43.523862 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:26:43.535964 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.526226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.0.98.62:33741)
0926 07:26:43.526344 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 07:26:43.526348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:26:43.526807 (+ 459us) server_negotiation.cc:407] Connection header received
0926 07:26:43.526982 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:26:43.526985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:26:43.527068 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:26:43.527127 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:26:43.528072 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.528883 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.530083 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.530424 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.530491 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.532409 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:26:43.532413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:26:43.532416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:26:43.532441 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:26:43.534104 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:26:43.534603 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:26:43.534607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:26:43.534609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:26:43.534688 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:26:43.534974 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:26:43.534977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:26:43.534978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:26:43.535218 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:26:43.535320 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:26:43.535668 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 07:26:43.535784 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:26:43.539826 11654 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:26:43.544618 11654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:26:43.575074 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.558314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33399 (local address 127.0.0.1:55552)
0926 07:26:43.561388 (+ 3074us) negotiation.cc:107] Waiting for socket to connect
0926 07:26:43.561399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:26:43.561478 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:26:43.563659 (+ 2181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:26:43.563665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:26:43.563687 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:26:43.563808 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.563814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.565219 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.565223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.567569 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.567582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.574453 (+ 6871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.574460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.574602 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.574645 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 07:26:43.574745 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:26:43.574870 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3018,"thread_start_us":91,"threads_started":1}
I20260926 07:26:43.576897 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.555866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42381 (local address 127.0.0.1:40122)
0926 07:26:43.560347 (+ 4481us) negotiation.cc:107] Waiting for socket to connect
0926 07:26:43.560398 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:26:43.560521 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:26:43.565995 (+ 5474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:26:43.565998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:26:43.566015 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:26:43.566099 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.566104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.571951 (+ 5847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.571954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.573148 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.573163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.576509 (+ 3346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.576513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.576600 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.576623 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:26:43.576697 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:26:43.576763 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4303,"thread_start_us":97,"threads_started":1}
I20260926 07:26:43.577868 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.558688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.0.98.2:33399)
0926 07:26:43.561262 (+ 2574us) server_negotiation.cc:206] Beginning negotiation
0926 07:26:43.561273 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:26:43.561564 (+ 291us) server_negotiation.cc:407] Connection header received
0926 07:26:43.561645 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:26:43.561650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:26:43.561873 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:26:43.562032 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:26:43.564010 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.565084 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.569894 (+ 4810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.570753 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.570886 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.571061 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 07:26:43.577592 (+ 6531us) server_negotiation.cc:299] Negotiation successful
0926 07:26:43.577706 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2467,"thread_start_us":94,"threads_started":1}
I20260926 07:26:43.577953 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.559453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44293 (local address 127.0.0.1:57876)
0926 07:26:43.567218 (+ 7765us) negotiation.cc:107] Waiting for socket to connect
0926 07:26:43.567234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:26:43.567331 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:26:43.571728 (+ 4397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:26:43.571731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:26:43.571752 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:26:43.571881 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.571888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.574802 (+ 2914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.574806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.576167 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.576179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.577465 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.577469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.577587 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.577628 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 07:26:43.577703 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:26:43.577794 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7696,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260926 07:26:43.578868 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.559891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.0.98.1:44293)
0926 07:26:43.571268 (+ 11377us) server_negotiation.cc:206] Beginning negotiation
0926 07:26:43.571275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:26:43.571300 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:26:43.571376 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:26:43.571381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:26:43.571587 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:26:43.571752 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:26:43.573350 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.574659 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.576339 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.577299 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.578410 (+ 1111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.578562 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:26:43.578629 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 07:26:43.578724 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11264,"thread_start_us":80,"threads_started":1}
I20260926 07:26:43.581385 11922 tablet_service.cc:1467] Processing CreateTablet for tablet c490f7a6bb8a4fab86ce8621a5937023 (DEFAULT_TABLE table=test-table [id=a2dbcb262fa5460fb17a3f972a5568b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:26:43.581934 11802 tablet_service.cc:1467] Processing CreateTablet for tablet c490f7a6bb8a4fab86ce8621a5937023 (DEFAULT_TABLE table=test-table [id=a2dbcb262fa5460fb17a3f972a5568b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:26:43.582599 11922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c490f7a6bb8a4fab86ce8621a5937023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:26:43.582947 11802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c490f7a6bb8a4fab86ce8621a5937023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:26:43.591253 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.556831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.0.98.3:42381)
0926 07:26:43.562996 (+ 6165us) server_negotiation.cc:206] Beginning negotiation
0926 07:26:43.563006 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:26:43.563047 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:26:43.563128 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:26:43.563133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:26:43.563326 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:26:43.563467 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:26:43.567723 (+ 4256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.568906 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.575383 (+ 6477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.576192 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.576300 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.576441 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:26:43.590907 (+ 14466us) server_negotiation.cc:299] Negotiation successful
0926 07:26:43.591079 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6062,"thread_start_us":153,"threads_started":1}
I20260926 07:26:43.594280 12072 tablet_service.cc:1467] Processing CreateTablet for tablet c490f7a6bb8a4fab86ce8621a5937023 (DEFAULT_TABLE table=test-table [id=a2dbcb262fa5460fb17a3f972a5568b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:26:43.595384 12072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c490f7a6bb8a4fab86ce8621a5937023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:26:43.597880 12158 tablet_bootstrap.cc:492] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342: Bootstrap starting.
I20260926 07:26:43.598376 12157 tablet_bootstrap.cc:492] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0: Bootstrap starting.
I20260926 07:26:43.600250 12158 tablet_bootstrap.cc:654] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342: Neither blocks nor log segments found. Creating new log.
I20260926 07:26:43.601583 12158 log.cc:824] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342: Log is configured to *not* fsync() on all Append() calls
I20260926 07:26:43.602427 12157 tablet_bootstrap.cc:654] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0: Neither blocks nor log segments found. Creating new log.
I20260926 07:26:43.603653 12158 tablet_bootstrap.cc:492] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342: No bootstrap required, opened a new log
I20260926 07:26:43.603842 12158 ts_tablet_manager.cc:1397] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:26:43.604604 12157 log.cc:824] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:26:43.604707 12160 tablet_bootstrap.cc:492] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2: Bootstrap starting.
I20260926 07:26:43.607270 12160 tablet_bootstrap.cc:654] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2: Neither blocks nor log segments found. Creating new log.
I20260926 07:26:43.607999 12160 log.cc:824] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:26:43.607882 12158 raft_consensus.cc:348] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd209a8ad834b4099896c4ecc7b7fd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "84500f2208004152a192bedcfd5a8342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33399 } } peers { permanent_uuid: "3fc5db53ecf34ca6926b4f1852055ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44293 } }
I20260926 07:26:43.608147 12158 raft_consensus.cc:374] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:26:43.608207 12158 raft_consensus.cc:729] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84500f2208004152a192bedcfd5a8342, State: Initialized, Role: FOLLOWER
I20260926 07:26:43.608707 12158 consensus_queue.cc:260] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd209a8ad834b4099896c4ecc7b7fd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "84500f2208004152a192bedcfd5a8342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33399 } } peers { permanent_uuid: "3fc5db53ecf34ca6926b4f1852055ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44293 } }
I20260926 07:26:43.609715 12158 ts_tablet_manager.cc:1428] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260926 07:26:43.610045 12004 heartbeater.cc:503] Master 127.0.98.62:33741 was elected leader, sending a full tablet report...
W20260926 07:26:43.611598 12005 tablet.cc:2369] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:26:43.613246 12157 tablet_bootstrap.cc:492] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0: No bootstrap required, opened a new log
I20260926 07:26:43.613472 12157 ts_tablet_manager.cc:1397] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260926 07:26:43.611016 12160 tablet_bootstrap.cc:492] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2: No bootstrap required, opened a new log
I20260926 07:26:43.614307 12160 ts_tablet_manager.cc:1397] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20260926 07:26:43.618227 12160 raft_consensus.cc:348] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd209a8ad834b4099896c4ecc7b7fd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "84500f2208004152a192bedcfd5a8342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33399 } } peers { permanent_uuid: "3fc5db53ecf34ca6926b4f1852055ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44293 } }
I20260926 07:26:43.618486 12160 raft_consensus.cc:374] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:26:43.618539 12160 raft_consensus.cc:729] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd209a8ad834b4099896c4ecc7b7fd2, State: Initialized, Role: FOLLOWER
I20260926 07:26:43.618561 12157 raft_consensus.cc:348] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd209a8ad834b4099896c4ecc7b7fd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "84500f2208004152a192bedcfd5a8342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33399 } } peers { permanent_uuid: "3fc5db53ecf34ca6926b4f1852055ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44293 } }
I20260926 07:26:43.618810 12157 raft_consensus.cc:374] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:26:43.618870 12157 raft_consensus.cc:729] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc5db53ecf34ca6926b4f1852055ee0, State: Initialized, Role: FOLLOWER
I20260926 07:26:43.619324 12157 consensus_queue.cc:260] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd209a8ad834b4099896c4ecc7b7fd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "84500f2208004152a192bedcfd5a8342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33399 } } peers { permanent_uuid: "3fc5db53ecf34ca6926b4f1852055ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44293 } }
I20260926 07:26:43.620155 12160 consensus_queue.cc:260] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd209a8ad834b4099896c4ecc7b7fd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "84500f2208004152a192bedcfd5a8342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33399 } } peers { permanent_uuid: "3fc5db53ecf34ca6926b4f1852055ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44293 } }
I20260926 07:26:43.620368 12157 ts_tablet_manager.cc:1428] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 07:26:43.620774 11868 heartbeater.cc:503] Master 127.0.98.62:33741 was elected leader, sending a full tablet report...
I20260926 07:26:43.621104 12160 ts_tablet_manager.cc:1428] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260926 07:26:43.622327 12138 heartbeater.cc:503] Master 127.0.98.62:33741 was elected leader, sending a full tablet report...
I20260926 07:26:43.644908 12163 raft_consensus.cc:482] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:26:43.645126 12163 raft_consensus.cc:504] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd209a8ad834b4099896c4ecc7b7fd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "84500f2208004152a192bedcfd5a8342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33399 } } peers { permanent_uuid: "3fc5db53ecf34ca6926b4f1852055ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44293 } }
I20260926 07:26:43.646332 12163 leader_election.cc:288] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dd209a8ad834b4099896c4ecc7b7fd2 (127.0.98.3:42381), 3fc5db53ecf34ca6926b4f1852055ee0 (127.0.98.1:44293)
I20260926 07:26:43.651640 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.647128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56551 (local address 127.0.98.1:44293)
0926 07:26:43.647272 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:26:43.647277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:26:43.647450 (+ 173us) server_negotiation.cc:407] Connection header received
0926 07:26:43.647627 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:26:43.647630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:26:43.647680 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:26:43.647728 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:26:43.648106 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.648804 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.650285 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.650961 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.651023 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.651104 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 07:26:43.651476 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 07:26:43.651545 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:26:43.651648 12006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.646478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42381 (local address 127.0.98.2:39897)
0926 07:26:43.646766 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 07:26:43.646784 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:26:43.646873 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:26:43.647550 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:26:43.647556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:26:43.647572 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:26:43.647707 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.647717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.648782 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.648786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.650145 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.650158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.651145 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.651149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.651286 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.651358 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 07:26:43.651432 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:26:43.651506 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":81}
I20260926 07:26:43.652026 12166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.646606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44293 (local address 127.0.98.2:56551)
0926 07:26:43.647374 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0926 07:26:43.647387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:26:43.647513 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:26:43.647864 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:26:43.647867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:26:43.647883 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:26:43.647977 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.647982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.648918 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.648921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.650168 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:26:43.650174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.651188 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.651191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:26:43.651310 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.651371 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 07:26:43.651816 (+ 445us) client_negotiation.cc:240] Negotiation successful
0926 07:26:43.651893 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":111,"thread_start_us":169,"threads_started":1}
I20260926 07:26:43.652555 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.647050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39897 (local address 127.0.98.3:42381)
0926 07:26:43.647167 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:26:43.647172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:26:43.647187 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:26:43.647239 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:26:43.647242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:26:43.647289 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:26:43.647362 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:26:43.647832 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.648638 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.650284 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.650888 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.650994 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.651086 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:26:43.652366 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0926 07:26:43.652469 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:26:43.652683 11822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c490f7a6bb8a4fab86ce8621a5937023" candidate_uuid: "84500f2208004152a192bedcfd5a8342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5db53ecf34ca6926b4f1852055ee0" is_pre_election: true
I20260926 07:26:43.653051 11822 raft_consensus.cc:2454] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84500f2208004152a192bedcfd5a8342 in term 0.
I20260926 07:26:43.653187 12092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c490f7a6bb8a4fab86ce8621a5937023" candidate_uuid: "84500f2208004152a192bedcfd5a8342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd209a8ad834b4099896c4ecc7b7fd2" is_pre_election: true
I20260926 07:26:43.653478 12092 raft_consensus.cc:2454] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84500f2208004152a192bedcfd5a8342 in term 0.
I20260926 07:26:43.653951 11894 leader_election.cc:302] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc5db53ecf34ca6926b4f1852055ee0, 84500f2208004152a192bedcfd5a8342; no voters:
I20260926 07:26:43.654218 12163 raft_consensus.cc:2781] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:26:43.654294 12163 raft_consensus.cc:482] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:26:43.654335 12163 raft_consensus.cc:3037] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:26:43.655478 12163 raft_consensus.cc:504] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd209a8ad834b4099896c4ecc7b7fd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "84500f2208004152a192bedcfd5a8342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33399 } } peers { permanent_uuid: "3fc5db53ecf34ca6926b4f1852055ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44293 } }
I20260926 07:26:43.655856 12163 leader_election.cc:288] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [CANDIDATE]: Term 1 election: Requested vote from peers 8dd209a8ad834b4099896c4ecc7b7fd2 (127.0.98.3:42381), 3fc5db53ecf34ca6926b4f1852055ee0 (127.0.98.1:44293)
I20260926 07:26:43.656175 12092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c490f7a6bb8a4fab86ce8621a5937023" candidate_uuid: "84500f2208004152a192bedcfd5a8342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd209a8ad834b4099896c4ecc7b7fd2"
I20260926 07:26:43.656260 11822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c490f7a6bb8a4fab86ce8621a5937023" candidate_uuid: "84500f2208004152a192bedcfd5a8342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc5db53ecf34ca6926b4f1852055ee0"
I20260926 07:26:43.656291 12092 raft_consensus.cc:3037] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:26:43.656366 11822 raft_consensus.cc:3037] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:26:43.657536 12092 raft_consensus.cc:2454] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84500f2208004152a192bedcfd5a8342 in term 1.
I20260926 07:26:43.657536 11822 raft_consensus.cc:2454] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84500f2208004152a192bedcfd5a8342 in term 1.
I20260926 07:26:43.657924 11891 leader_election.cc:302] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84500f2208004152a192bedcfd5a8342, 8dd209a8ad834b4099896c4ecc7b7fd2; no voters:
I20260926 07:26:43.658427 12163 raft_consensus.cc:2781] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:26:43.658653 12163 raft_consensus.cc:686] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [term 1 LEADER]: Becoming Leader. State: Replica: 84500f2208004152a192bedcfd5a8342, State: Running, Role: LEADER
I20260926 07:26:43.658999 12163 consensus_queue.cc:237] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd209a8ad834b4099896c4ecc7b7fd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "84500f2208004152a192bedcfd5a8342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33399 } } peers { permanent_uuid: "3fc5db53ecf34ca6926b4f1852055ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44293 } }
I20260926 07:26:43.662638 11653 catalog_manager.cc:5495] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84500f2208004152a192bedcfd5a8342 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "84500f2208004152a192bedcfd5a8342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd209a8ad834b4099896c4ecc7b7fd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84500f2208004152a192bedcfd5a8342" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fc5db53ecf34ca6926b4f1852055ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44293 } health_report { overall_health: UNKNOWN } } }
Sep 26 07:26:43 dist-test-slave-g04g krb5kdc[11178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:26:43.696619 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.691146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.0.98.2:33399)
0926 07:26:43.691287 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:26:43.691291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:26:43.691305 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:26:43.691391 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:26:43.691393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:26:43.691451 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:26:43.691576 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:26:43.691877 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.692776 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.694215 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:26:43.694571 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:26:43.694694 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:26:43.695175 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:26:43.695839 (+ 664us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:26:43.695954 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:26:43.696371 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 07:26:43.696433 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:26:43.707315 11822 raft_consensus.cc:1261] T c490f7a6bb8a4fab86ce8621a5937023 P 3fc5db53ecf34ca6926b4f1852055ee0 [term 1 FOLLOWER]: Refusing update from remote peer 84500f2208004152a192bedcfd5a8342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:26:43.708163 12163 consensus_queue.cc:1035] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc5db53ecf34ca6926b4f1852055ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:26:43.713135 12092 raft_consensus.cc:1261] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2 [term 1 FOLLOWER]: Refusing update from remote peer 84500f2208004152a192bedcfd5a8342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:26:43.713804 12163 consensus_queue.cc:1035] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dd209a8ad834b4099896c4ecc7b7fd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:26:43.732777 12171 mvcc.cc:205] Tried to move back new op lower bound from 7333509544776830976 to 7333509544590139392. Current Snapshot: MvccSnapshot[applied={T|T < 7333509544776830976 or (T in {7333509544776830976})}]
I20260926 07:26:43.735603 12174 mvcc.cc:205] Tried to move back new op lower bound from 7333509544776830976 to 7333509544590139392. Current Snapshot: MvccSnapshot[applied={T|T < 7333509544776830976}]
I20260926 07:26:43.738701 11653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:26:43.738873 11653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:26:43.741683 11653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0613bf6ec5fb497dbc11373ed1d216c2: Sending DeleteTablet for 3 replicas of tablet c490f7a6bb8a4fab86ce8621a5937023
I20260926 07:26:43.742710 12072 tablet_service.cc:1514] Processing DeleteTablet for tablet c490f7a6bb8a4fab86ce8621a5937023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:26:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40122
I20260926 07:26:43.742851 11922 tablet_service.cc:1514] Processing DeleteTablet for tablet c490f7a6bb8a4fab86ce8621a5937023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:26:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55552
I20260926 07:26:43.742962 11802 tablet_service.cc:1514] Processing DeleteTablet for tablet c490f7a6bb8a4fab86ce8621a5937023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:26:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57876
I20260926 07:26:43.744423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 11735
I20260926 07:26:43.757974 12181 tablet_replica.cc:330] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342: stopping tablet replica
I20260926 07:26:43.758386 12181 raft_consensus.cc:2229] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:26:43.759222 12181 raft_consensus.cc:2258] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:26:43.760993 12181 ts_tablet_manager.cc:1905] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:26:43.761291 12180 tablet_replica.cc:330] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2: stopping tablet replica
I20260926 07:26:43.761658 12180 raft_consensus.cc:2229] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 07:26:43.762143 11641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44293 (error 108)
I20260926 07:26:43.765377 12180 raft_consensus.cc:2258] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:26:43.771417 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.771051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44293 (local address 127.0.0.1:57888)
0926 07:26:43.771249 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 07:26:43.771326 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260926 07:26:43.771729 11641 catalog_manager.cc:4584] TS 3fc5db53ecf34ca6926b4f1852055ee0 (127.0.98.1:44293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c490f7a6bb8a4fab86ce8621a5937023: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44293: connect: Connection refused (error 111)
I20260926 07:26:43.774765 12181 ts_tablet_manager.cc:1918] T c490f7a6bb8a4fab86ce8621a5937023 P 84500f2208004152a192bedcfd5a8342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:26:43.775177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 11874
I20260926 07:26:43.780592 12180 ts_tablet_manager.cc:1905] T c490f7a6bb8a4fab86ce8621a5937023 P 8dd209a8ad834b4099896c4ecc7b7fd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:26:43.783936 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 12009
I20260926 07:26:43.785593 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:26:43.785205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33399 (local address 127.0.0.1:55568)
0926 07:26:43.785417 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 07:26:43.785481 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 07:26:43.791931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 11622
I20260926 07:26:43.801793 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 07:26:44.146373 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 07:26:44.146533 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 07:26:44.490881 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 07:26:44.491 UTC [11208] LOG: received smart shutdown request
2026-09-26 07:26:44.494 UTC [11208] LOG: background worker "logical replication launcher" (PID 11217) exited with exit code 1
2026-09-26 07:26:44.495 UTC [11212] LOG: shutting down
2026-09-26 07:26:44.672 UTC [11208] LOG: database system is shut down
2026-09-26T07:26:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47442 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:26:44 dist-test-slave-g04g krb5kdc[12193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:26:44 dist-test-slave-g04g krb5kdc[12193](info): set up 2 sockets
Sep 26 07:26:44 dist-test-slave-g04g krb5kdc[12193](info): commencing operation
krb5kdc: starting...
W20260926 07:26:45.839224 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:26:45 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:26:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:26:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 07:26:46.007740 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_EJaZL/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/postgres -l logfile start
2026-09-26 07:26:48.606 UTC [12223] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 07:26:48.606 UTC [12223] LOG: listening on IPv4 address "127.0.98.20", port 48725
2026-09-26 07:26:48.610 UTC [12223] LOG: listening on Unix socket "/tmp/.s.PGSQL.48725"
2026-09-26 07:26:48.648 UTC [12226] LOG: database system was shut down at 2026-09-26 07:26:47 UTC
2026-09-26 07:26:48.653 UTC [12223] LOG: database system is ready to accept connections
I20260926 07:26:49.655169 392 mini_postgres.cc:96] Postgres bound to 48725
2026-09-26 07:26:49.661 UTC [12234] FATAL: database "slave" does not exist
127.0.98.20:48725 - accepting connections
I20260926 07:26:49.662189 392 mini_ranger.cc:161] Starting Ranger...
I20260926 07:26:49.678546 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 07:26:50.279503 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 07:26:50.279608 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-admin
2026-09-26 07:26:50,542 [I] DB FLAVOR :POSTGRES
2026-09-26 07:26:50,543 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:26:50,543 [I] Checking connection..
2026-09-26 07:26:50,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 07:26:50,819 [I] Checking connection passed.
2026-09-26 07:26:50,819 [I] --------- Verifying version history table ---------
2026-09-26 07:26:50,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:26:51,075 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:26:51,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:26:51,343 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:26:51,343 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 07:26:51,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 07:26:51.590 UTC [12298] WARNING: there is no transaction in progress
2026-09-26 07:26:51,597 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 07:26:51,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:26:51,845 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 07:26:51,845 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:26:51,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 07:26:52,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 07:26:52,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g04g.c.gcp-upstream.internal','N') ;"
2026-09-26 07:26:52,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:26:52,808 [I] Table x_portal_user does not exist in database ranger
2026-09-26 07:26:52,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:26:53,059 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 07:26:53,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:26:53,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:26:53,558 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 07:26:53,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 07:26:54.535 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.553 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.570 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.587 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.603 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.619 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.685 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.692 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.708 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.718 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.728 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.743 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.753 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.760 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.768 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:54.775 UTC [12406] WARNING: there is no transaction in progress
2026-09-26 07:26:55,462 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 07:26:55,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 07:26:55,693 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 07:26:55,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:26:55,930 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 07:26:55,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:26:56,195 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 07:26:56,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:26:56,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:26:56,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g04g.c.gcp-upstream.internal';"
2026-09-26 07:26:56,909 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 07:26:56,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 07:26:57,178 [I] DB_PATCHES have already been applied
I20260926 07:26:57.183840 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_EJaZL/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 07:26:57.183903 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 07:26:57.187283 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:43487
Sep 26, 2026 7:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:26:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:26:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:26:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 07:26:58.401464 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.739s user 0.003s sys 0.007s
Sep 26, 2026 7:27:18 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:27:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 7:27:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:27:19 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:27:19 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 07:27:20.572328 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 07:27:20 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407640, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 07:27:20.676003 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 07:27:20.686189 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 07:27:21.648608 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 07:27:21.648712 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-kms
I20260926 07:27:21.648736 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:48725
2026-09-26 07:27:21,814 [I] DB FLAVOR :POSTGRES
2026-09-26 07:27:21,814 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:27:21,814 [I] Checking connection
2026-09-26 07:27:21,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 07:27:22,050 [I] connection success
2026-09-26 07:27:22,051 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 07:27:22,051 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 07:27:22,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 07:27:22,284 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 07:27:22,285 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:27:22,285 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 07:27:22,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48725/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 07:27:22,537 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 07:27:22.993265 392 mini_ranger_kms.cc:310] Created kms service
I20260926 07:27:23.196821 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 07:27:23.239882 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 07:27:23 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407640, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:27:23.659976 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 07:27:23.660126 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task_EJaZL/build/debug/bin/postgresql.jar:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_EJaZL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 07:27:23.660148 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 07:27:23.663602 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12586
I20260926 07:27:23.663689 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:55027
Sep 26, 2026 7:27:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:27:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:27:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:27:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:27:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:27:25 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 07:27:25.928956 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.253s user 0.019s sys 0.131s
I20260926 07:27:25.929095 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 07:27:25.929201 392 mini_ranger_kms.cc:401] 127.0.98.20:55027/kms/v1/keys
Sep 26 07:27:26 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407646, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 07:27:26 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407646, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 07:27:26 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407646, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 07:27:29 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:27:29.150054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36957
--ranger_config_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55027/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:29.283255 12637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:29.283491 12637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:29.283528 12637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:29.287460 12637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:29.287511 12637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:29.287530 12637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:29.287546 12637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:29.287561 12637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:29.292972 12637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:55027/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36957
--ranger_config_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:29.293977 12637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:29.294958 12637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:29.303812 12643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:29.303995 12637 server_base.cc:1034] running on GCE node
W20260926 07:27:29.303853 12645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:29.304104 12642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:29.304692 12637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:29.305774 12637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:29.307016 12637 hybrid_clock.cc:648] HybridClock initialized: now 1790407649306996 us; error 37 us; skew 500 ppm
Sep 26 07:27:29 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:29.312845 12637 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:27:29.313750 12637 webserver.cc:460] Webserver started at http://127.0.98.62:38053/ using document root <none> and password file <none>
I20260926 07:27:29.314286 12637 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:29.314342 12637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:29.314554 12637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:27:29 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:27:29.370038 12637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "a658dbb5445a4a03a9e84573aa552014"
format_stamp: "Formatted at 2026-09-26 07:27:29 on dist-test-slave-g04g"
server_key: "6be7466f80b3473eb294f6224ee88c21"
server_key_iv: "a5e7a59a1a14a55805fca1eab142c24c"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:29.370749 12637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "a658dbb5445a4a03a9e84573aa552014"
format_stamp: "Formatted at 2026-09-26 07:27:29 on dist-test-slave-g04g"
server_key: "6be7466f80b3473eb294f6224ee88c21"
server_key_iv: "a5e7a59a1a14a55805fca1eab142c24c"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:29.375387 12637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 07:27:29.389241 12652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:29.390654 12637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:29.390796 12637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "a658dbb5445a4a03a9e84573aa552014"
format_stamp: "Formatted at 2026-09-26 07:27:29 on dist-test-slave-g04g"
server_key: "6be7466f80b3473eb294f6224ee88c21"
server_key_iv: "a5e7a59a1a14a55805fca1eab142c24c"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:29.391166 12637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:29.408423 12637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:29.411816 12637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:29.412029 12637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:29.420663 12637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36957
I20260926 07:27:29.420670 12704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36957 every 8 connection(s)
I20260926 07:27:29.421820 12637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:29.424793 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:29.427886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 12637
I20260926 07:27:29.428014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:27:29.428330 392 external_mini_cluster.cc:1428] Setting key 41cd6c45aa996d1498bedc0864c2a60b
I20260926 07:27:29.431779 12705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014: Bootstrap starting.
I20260926 07:27:29.434406 12705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:29.435305 12705 log.cc:824] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:29.438587 12705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014: No bootstrap required, opened a new log
Sep 26 07:27:29 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:29.443138 12705 raft_consensus.cc:348] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658dbb5445a4a03a9e84573aa552014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } }
I20260926 07:27:29.443430 12705 raft_consensus.cc:374] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:29.443523 12705 raft_consensus.cc:729] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a658dbb5445a4a03a9e84573aa552014, State: Initialized, Role: FOLLOWER
I20260926 07:27:29.444077 12705 consensus_queue.cc:260] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658dbb5445a4a03a9e84573aa552014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } }
I20260926 07:27:29.444244 12705 raft_consensus.cc:388] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:29.444336 12705 raft_consensus.cc:482] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:29.444443 12705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:29.445652 12705 raft_consensus.cc:504] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658dbb5445a4a03a9e84573aa552014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } }
I20260926 07:27:29.446039 12705 leader_election.cc:302] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a658dbb5445a4a03a9e84573aa552014; no voters:
I20260926 07:27:29.446377 12705 leader_election.cc:288] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:29.446472 12710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:29.446650 12708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:29.429982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.0.98.62:36957)
0926 07:27:29.430556 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:29.430576 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:29.430619 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:27:29.431571 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:29.431595 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:29.432027 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:29.432389 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:29.433302 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:29.434882 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:29.436122 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:29.436619 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:29.436815 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:29.440540 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:29.440549 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:29.440567 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:29.440601 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:29.442827 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:29.443356 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:29.443362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:29.443368 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:29.443516 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:29.443887 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:29.443890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:29.443892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:29.444378 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:29.444569 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:29.445104 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 07:27:29.445582 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":147,"threads_started":1}
I20260926 07:27:29.446777 12710 raft_consensus.cc:686] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 1 LEADER]: Becoming Leader. State: Replica: a658dbb5445a4a03a9e84573aa552014, State: Running, Role: LEADER
I20260926 07:27:29.447304 12710 consensus_queue.cc:237] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658dbb5445a4a03a9e84573aa552014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } }
I20260926 07:27:29.447858 12705 sys_catalog.cc:564] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:29.448518 12710 sys_catalog.cc:455] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a658dbb5445a4a03a9e84573aa552014. Latest consensus state: current_term: 1 leader_uuid: "a658dbb5445a4a03a9e84573aa552014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658dbb5445a4a03a9e84573aa552014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } }
I20260926 07:27:29.448642 12710 sys_catalog.cc:458] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:29.449465 12712 sys_catalog.cc:455] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a658dbb5445a4a03a9e84573aa552014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658dbb5445a4a03a9e84573aa552014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } }
I20260926 07:27:29.450117 12712 sys_catalog.cc:458] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:29.450475 12705 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:27:29.450696 12716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:29.456594 12716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:29.465916 12716 catalog_manager.cc:1347] Generated new cluster ID: 775b06c574f6419386df643f89687d35
I20260926 07:27:29.465981 12716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:29.513640 12716 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:29.514981 12716 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:29.526599 12716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014: Generated new TSK 0
I20260926 07:27:29.527803 12716 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:27:30.164338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55027/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36957
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:30.328436 12750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:30.328805 12750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:30.328874 12750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:30.333460 12750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:30.333523 12750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:30.333616 12750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:27:30.339409 12750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:55027/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:30.340515 12750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:30.341630 12750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:30.353523 12758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:30.353969 12757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:30.354826 12760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:30.355942 12750 server_base.cc:1034] running on GCE node
I20260926 07:27:30.356415 12750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:30.357216 12750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:30.358422 12750 hybrid_clock.cc:648] HybridClock initialized: now 1790407650358387 us; error 49 us; skew 500 ppm
Sep 26 07:27:30 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:30.361974 12750 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:27:30.363054 12750 webserver.cc:460] Webserver started at http://127.0.98.1:41867/ using document root <none> and password file <none>
I20260926 07:27:30.363708 12750 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:30.363796 12750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:30.364014 12750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:27:30 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:27:30.379670 12750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "47b3b25c809b4cab9191efe706735467"
format_stamp: "Formatted at 2026-09-26 07:27:30 on dist-test-slave-g04g"
server_key: "5124874caa28be58b9b1fee0919406e9"
server_key_iv: "ff1d0633bbee9c036776c82389543d16"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:30.380482 12750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "47b3b25c809b4cab9191efe706735467"
format_stamp: "Formatted at 2026-09-26 07:27:30 on dist-test-slave-g04g"
server_key: "5124874caa28be58b9b1fee0919406e9"
server_key_iv: "ff1d0633bbee9c036776c82389543d16"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:30.385535 12750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 07:27:30.409562 12769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:30.411800 12750 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260926 07:27:30.411950 12750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "47b3b25c809b4cab9191efe706735467"
format_stamp: "Formatted at 2026-09-26 07:27:30 on dist-test-slave-g04g"
server_key: "5124874caa28be58b9b1fee0919406e9"
server_key_iv: "ff1d0633bbee9c036776c82389543d16"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:30.412109 12750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:30.433543 12750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:30.437537 12750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:30.438158 12750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:30.438971 12750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:30.440158 12750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:30.440271 12750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:30.440409 12750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:30.440498 12750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:30.454372 12750 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46603
I20260926 07:27:30.455732 12750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:30.464450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 12750
I20260926 07:27:30.464576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:27:30.464883 392 external_mini_cluster.cc:1428] Setting key 7b0ead6680029472939bd4cabbbe2cc3
Sep 26 07:27:30 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:30.471225 12882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46603 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 26 07:27:30 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 07:27:30.494839 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:30.458898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36957 (local address 127.0.98.1:42369)
0926 07:27:30.461820 (+ 2922us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:30.461884 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:30.463063 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:30.472167 (+ 9104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:30.472185 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:30.472604 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:30.473346 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:30.473370 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:30.474582 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:30.474586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:30.476021 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:30.476033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:30.476913 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:30.476918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:30.477225 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:30.477991 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:30.478016 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:30.488728 (+ 10712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:30.491203 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:30.491211 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:30.491232 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:30.491621 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:30.491953 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:30.491956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:30.491958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:30.492156 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:30.492647 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:30.492655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:30.493037 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 07:27:30.493254 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 07:27:30.493621 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2682,"thread_start_us":140,"threads_started":1}
I20260926 07:27:30.495460 12886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:30.460057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42369 (local address 127.0.98.62:36957)
0926 07:27:30.471537 (+ 11480us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:30.471548 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:30.471566 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:27:30.471617 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:30.471620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:30.471676 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:30.471790 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:30.473550 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:30.474404 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:30.476205 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:30.476574 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:30.476694 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:30.489028 (+ 12334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:30.489033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:30.489036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:30.489069 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:30.490963 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:30.491754 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:30.491757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:30.491760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:30.491855 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:30.492273 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:30.492276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:30.492278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:30.492508 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:30.492593 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:30.495148 (+ 2555us) server_negotiation.cc:299] Negotiation successful
0926 07:27:30.495285 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11384,"thread_start_us":137,"threads_started":1}
I20260926 07:27:30.497221 12883 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36957
I20260926 07:27:30.497498 12883 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:30.498104 12883 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
I20260926 07:27:30.499931 12669 ts_manager.cc:194] Registered new tserver with Master: 47b3b25c809b4cab9191efe706735467 (127.0.98.1:46603)
I20260926 07:27:30.501438 12669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:27:30.523255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55027/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36957
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:30.686864 12889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:30.687368 12889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:30.687505 12889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:30.693711 12889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:30.693845 12889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:30.693990 12889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:27:30.702298 12889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:55027/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:30.703935 12889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:30.705436 12889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:30.718683 12898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:30.718683 12896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:30.718986 12895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:30.720245 12889 server_base.cc:1034] running on GCE node
I20260926 07:27:30.720691 12889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:30.721489 12889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:30.722649 12889 hybrid_clock.cc:648] HybridClock initialized: now 1790407650722582 us; error 84 us; skew 500 ppm
Sep 26 07:27:30 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:30.726127 12889 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:27:30.727152 12889 webserver.cc:460] Webserver started at http://127.0.98.2:33369/ using document root <none> and password file <none>
I20260926 07:27:30.727942 12889 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:30.728097 12889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:30.728420 12889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:27:30 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:27:30.750259 12889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "5ff850e18da946fe92629e53cd48a05d"
format_stamp: "Formatted at 2026-09-26 07:27:30 on dist-test-slave-g04g"
server_key: "5a03fd444e3f0c4ac46734ef00bebf9b"
server_key_iv: "f421c8a8a746f5a7f10f7a1e80109d06"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:30.751405 12889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5ff850e18da946fe92629e53cd48a05d"
format_stamp: "Formatted at 2026-09-26 07:27:30 on dist-test-slave-g04g"
server_key: "5a03fd444e3f0c4ac46734ef00bebf9b"
server_key_iv: "f421c8a8a746f5a7f10f7a1e80109d06"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:30.756795 12889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 07:27:30.781018 12905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:30.782891 12889 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260926 07:27:30.783185 12889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "5ff850e18da946fe92629e53cd48a05d"
format_stamp: "Formatted at 2026-09-26 07:27:30 on dist-test-slave-g04g"
server_key: "5a03fd444e3f0c4ac46734ef00bebf9b"
server_key_iv: "f421c8a8a746f5a7f10f7a1e80109d06"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:30.783357 12889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:30.825585 12889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:30.829636 12889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:30.829898 12889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:30.830808 12889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:30.832226 12889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:30.832309 12889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:30.832377 12889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:30.832422 12889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:30.864722 12889 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38901
I20260926 07:27:30.866081 12889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:30.867800 13018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38901 every 8 connection(s)
I20260926 07:27:30.871661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 12889
I20260926 07:27:30.871812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:27:30.872195 392 external_mini_cluster.cc:1428] Setting key 7029d76e64152660ee4d1ec52a9495b1
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 26 07:27:30 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:30.902904 12886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:30.871329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55573 (local address 127.0.98.62:36957)
0926 07:27:30.871465 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:30.871470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:30.881045 (+ 9575us) server_negotiation.cc:407] Connection header received
0926 07:27:30.884676 (+ 3631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:30.884680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:30.884751 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:30.884857 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:30.886694 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:30.887768 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:30.891977 (+ 4209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:30.892376 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:30.892472 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:30.897408 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:30.897413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:30.897416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:30.897451 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:30.899482 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:30.900175 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:30.900179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:30.900181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:30.900277 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:30.901350 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:30.901354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:30.901356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:30.901601 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:30.901698 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:30.902571 (+ 873us) server_negotiation.cc:299] Negotiation successful
0926 07:27:30.902718 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:27:30.905421 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:30.870202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36957 (local address 127.0.98.2:55573)
0926 07:27:30.880765 (+ 10563us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:30.880915 (+ 150us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:30.882333 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:30.885085 (+ 2752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:30.885097 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:30.885542 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:30.886333 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:30.886357 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:30.887957 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:30.887961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:30.891794 (+ 3833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:30.891808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:30.892722 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:30.892727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:30.892952 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:30.894843 (+ 1891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:30.894869 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:30.897182 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:30.899642 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:30.899649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:30.899667 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:30.900042 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:30.900400 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:30.900403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:30.900406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:30.900607 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:30.901778 (+ 1171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:30.901788 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:30.902178 (+ 390us) client_negotiation.cc:769] Sending connection context
0926 07:27:30.903484 (+ 1306us) client_negotiation.cc:240] Negotiation successful
0926 07:27:30.903863 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10239,"thread_start_us":132,"threads_started":1}
I20260926 07:27:30.907239 13019 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36957
I20260926 07:27:30.907539 13019 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:30.908150 13019 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
I20260926 07:27:30.909390 12669 ts_manager.cc:194] Registered new tserver with Master: 5ff850e18da946fe92629e53cd48a05d (127.0.98.2:38901)
I20260926 07:27:30.910120 12669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:27:30.913949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55027/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36957
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:27:31 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 07:27:31.049305 13024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:31.049734 13024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:31.049853 13024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:31.055871 13024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:31.055948 13024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:31.056034 13024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:27:31.063668 13024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:55027/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:31.065179 13024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:31.066530 13024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:31.079471 13029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:31.079439 13030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:31.079988 13032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:31.081001 13024 server_base.cc:1034] running on GCE node
I20260926 07:27:31.081439 13024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:31.082172 13024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:31.085425 13024 hybrid_clock.cc:648] HybridClock initialized: now 1790407651085354 us; error 78 us; skew 500 ppm
Sep 26 07:27:31 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:31.089025 13024 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:27:31.090226 13024 webserver.cc:460] Webserver started at http://127.0.98.3:35371/ using document root <none> and password file <none>
I20260926 07:27:31.091007 13024 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:31.091156 13024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:31.091426 13024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:27:31 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:27:31.121881 13024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "03f50bee1eb4454c82cced28ab8d5d33"
format_stamp: "Formatted at 2026-09-26 07:27:31 on dist-test-slave-g04g"
server_key: "fdfeed7ad7f716d563a2e1a96c449ca8"
server_key_iv: "c49b1669813e163e41a88571bcf3045a"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:31.123438 13024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "03f50bee1eb4454c82cced28ab8d5d33"
format_stamp: "Formatted at 2026-09-26 07:27:31 on dist-test-slave-g04g"
server_key: "fdfeed7ad7f716d563a2e1a96c449ca8"
server_key_iv: "c49b1669813e163e41a88571bcf3045a"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:31.134907 13024 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 07:27:31.159487 13039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:31.162000 13024 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260926 07:27:31.162156 13024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "03f50bee1eb4454c82cced28ab8d5d33"
format_stamp: "Formatted at 2026-09-26 07:27:31 on dist-test-slave-g04g"
server_key: "fdfeed7ad7f716d563a2e1a96c449ca8"
server_key_iv: "c49b1669813e163e41a88571bcf3045a"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:31.162305 13024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:31.184508 13024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:31.188308 13024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:31.188591 13024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:31.189320 13024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:31.190371 13024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:31.190428 13024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:31.190476 13024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:31.190541 13024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:31.200366 13024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42121
I20260926 07:27:31.200588 13152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42121 every 8 connection(s)
I20260926 07:27:31.201427 13024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:31.205572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 13024
I20260926 07:27:31.205677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:27:31.205963 392 external_mini_cluster.cc:1428] Setting key d7d4c750fddd3cff4988cb83466eb682
Sep 26 07:27:31 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:31.218219 12886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:31.203465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60351 (local address 127.0.98.62:36957)
0926 07:27:31.203593 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:31.203597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:31.204668 (+ 1071us) server_negotiation.cc:407] Connection header received
0926 07:27:31.206145 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:31.206149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:31.206209 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:31.206269 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:31.207913 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:31.208737 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:31.210465 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:31.210811 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:31.210922 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:31.213826 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:31.213830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:31.213833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:31.213860 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:31.215684 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:31.216299 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:31.216303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:31.216305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:31.216399 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:31.216875 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:31.216879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:31.216881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:31.217132 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:31.217243 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:31.217894 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 07:27:31.218031 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:27:31.219950 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:31.203952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36957 (local address 127.0.98.3:60351)
0926 07:27:31.204505 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:31.204559 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:31.205819 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:31.206680 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:31.206689 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:31.207024 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:31.207731 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:31.207754 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:31.208905 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:31.208909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:31.210328 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:31.210338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:31.210976 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:31.210980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:31.211218 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:31.211896 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:31.211917 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:31.213641 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:31.215810 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:31.215816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:31.215832 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:31.216169 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:31.216509 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:31.216512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:31.216514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:31.216697 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:31.217249 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:31.217256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:31.217607 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 07:27:31.217820 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 07:27:31.218829 (+ 1009us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":145,"threads_started":1}
I20260926 07:27:31.221091 13153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36957
I20260926 07:27:31.221362 13153 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:31.221942 13153 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
I20260926 07:27:31.223109 12669 ts_manager.cc:194] Registered new tserver with Master: 03f50bee1eb4454c82cced28ab8d5d33 (127.0.98.3:42121)
I20260926 07:27:31.223810 12669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60351
I20260926 07:27:31.231559 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:27:31.232349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 12750
I20260926 07:27:31.240722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 12889
I20260926 07:27:31.248435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 13024
I20260926 07:27:31.255610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 12637
I20260926 07:27:31.263695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36957
--webserver_interface=127.0.98.62
--webserver_port=38053
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36957
--ranger_config_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55027/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55027/kms with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:31.374378 13156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:31.374660 13156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:31.374701 13156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:31.378693 13156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:31.378753 13156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:31.378772 13156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:31.378788 13156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:31.378803 13156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:31.384251 13156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55027/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36957
--ranger_config_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:31.385185 13156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:31.386124 13156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:31.394608 13162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:31.394733 13164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:31.394615 13161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:31.394654 13156 server_base.cc:1034] running on GCE node
I20260926 07:27:31.395383 13156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:31.396525 13156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:31.397704 13156 hybrid_clock.cc:648] HybridClock initialized: now 1790407651397677 us; error 40 us; skew 500 ppm
Sep 26 07:27:31 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:31.400521 13156 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:27:31.401348 13156 webserver.cc:460] Webserver started at http://127.0.98.62:38053/ using document root <none> and password file <none>
I20260926 07:27:31.401870 13156 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:31.401962 13156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:27:31 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:27:31.463441 13156 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:31.465840 13172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:31.467315 13156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:31.467460 13156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "a658dbb5445a4a03a9e84573aa552014"
format_stamp: "Formatted at 2026-09-26 07:27:29 on dist-test-slave-g04g"
server_key: "6be7466f80b3473eb294f6224ee88c21"
server_key_iv: "a5e7a59a1a14a55805fca1eab142c24c"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:31.468037 13156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:31.503139 13156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:31.506229 13156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:31.506444 13156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:31.513928 13156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36957
I20260926 07:27:31.513963 13224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36957 every 8 connection(s)
I20260926 07:27:31.514972 13156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:31.519115 13225 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:27:31.520712 13225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014: Bootstrap starting.
I20260926 07:27:31.521622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 13156
I20260926 07:27:31.522135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55027/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46603
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36957
--webserver_port=41867
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55027/kms with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:27:31.542889 13225 log.cc:824] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:31.551097 13225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:27:31.551535 13225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014: Bootstrap complete.
I20260926 07:27:31.556922 13225 raft_consensus.cc:348] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658dbb5445a4a03a9e84573aa552014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } }
I20260926 07:27:31.557435 13225 raft_consensus.cc:729] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a658dbb5445a4a03a9e84573aa552014, State: Initialized, Role: FOLLOWER
I20260926 07:27:31.557987 13225 consensus_queue.cc:260] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658dbb5445a4a03a9e84573aa552014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } }
I20260926 07:27:31.558135 13225 raft_consensus.cc:388] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:31.558224 13225 raft_consensus.cc:482] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:31.558326 13225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:27:31.560725 13225 raft_consensus.cc:504] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658dbb5445a4a03a9e84573aa552014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } }
I20260926 07:27:31.561081 13225 leader_election.cc:302] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a658dbb5445a4a03a9e84573aa552014; no voters:
I20260926 07:27:31.561406 13225 leader_election.cc:288] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:27:31.561519 13229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:27:31.561803 13229 raft_consensus.cc:686] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [term 2 LEADER]: Becoming Leader. State: Replica: a658dbb5445a4a03a9e84573aa552014, State: Running, Role: LEADER
I20260926 07:27:31.562156 13229 consensus_queue.cc:237] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658dbb5445a4a03a9e84573aa552014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } }
I20260926 07:27:31.562662 13225 sys_catalog.cc:564] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:31.563155 13229 sys_catalog.cc:455] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a658dbb5445a4a03a9e84573aa552014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658dbb5445a4a03a9e84573aa552014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } }
I20260926 07:27:31.563338 13229 sys_catalog.cc:458] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:31.563580 13229 sys_catalog.cc:455] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a658dbb5445a4a03a9e84573aa552014. Latest consensus state: current_term: 2 leader_uuid: "a658dbb5445a4a03a9e84573aa552014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a658dbb5445a4a03a9e84573aa552014" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36957 } } }
I20260926 07:27:31.563715 13229 sys_catalog.cc:458] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:31.565001 13225 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:27:31.565421 13234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:31.572083 13234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:31.573179 13234 catalog_manager.cc:1259] Loaded cluster ID: 775b06c574f6419386df643f89687d35
I20260926 07:27:31.573297 13234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:31.576635 13234 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:31.577526 13234 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014: Loaded TSK: 0
I20260926 07:27:31.578907 13234 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:27:31.703678 13227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:31.704131 13227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:31.704263 13227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:31.711220 13227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:31.711359 13227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:31.711503 13227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:27:31.719563 13227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55027/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41867
--tserver_master_addrs=127.0.98.62:36957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:31.721087 13227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:31.722455 13227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:31.735469 13263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:31.736052 13262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:31.737133 13227 server_base.cc:1034] running on GCE node
W20260926 07:27:31.737344 13265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:31.737874 13227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:31.738667 13227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:31.739851 13227 hybrid_clock.cc:648] HybridClock initialized: now 1790407651739802 us; error 64 us; skew 500 ppm
Sep 26 07:27:31 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:31.743922 13227 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:27:31.745042 13227 webserver.cc:460] Webserver started at http://127.0.98.1:41867/ using document root <none> and password file <none>
I20260926 07:27:31.745743 13227 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:31.745831 13227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:27:31 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:27:31.780720 13227 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260926 07:27:31.795022 13273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:31.799528 13227 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.003s sys 0.000s
I20260926 07:27:31.799712 13227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "47b3b25c809b4cab9191efe706735467"
format_stamp: "Formatted at 2026-09-26 07:27:30 on dist-test-slave-g04g"
server_key: "5124874caa28be58b9b1fee0919406e9"
server_key_iv: "ff1d0633bbee9c036776c82389543d16"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:31.800398 13227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:31.813115 13227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:31.816789 13227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:31.817044 13227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:31.817763 13227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:31.818807 13227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:31.818910 13227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:31.819062 13227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:31.819150 13227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:31.835075 13227 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46603
I20260926 07:27:31.835114 13386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46603 every 8 connection(s)
I20260926 07:27:31.836158 13227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:31.841171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 13227
I20260926 07:27:31.841677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55027/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38901
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36957
--webserver_port=33369
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55027/kms with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:27:31 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:31.868156 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:31.838705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36957 (local address 127.0.98.1:47195)
0926 07:27:31.841190 (+ 2485us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:31.841330 (+ 140us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:31.842915 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:31.844899 (+ 1984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:31.844907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:31.845253 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:31.845887 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:31.845906 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:31.847752 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:31.847757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:31.849130 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:31.849143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:31.850533 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:31.850539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:31.850755 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:31.859254 (+ 8499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:31.859283 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:31.861165 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:31.864108 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:31.864116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:31.864133 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:31.864527 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:31.864989 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:31.864992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:31.864994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:31.865177 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:31.866041 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:31.866048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:31.866393 (+ 345us) client_negotiation.cc:769] Sending connection context
0926 07:27:31.866629 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 07:27:31.866995 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2259,"thread_start_us":140,"threads_started":1}
I20260926 07:27:31.869890 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:31.839601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47195 (local address 127.0.98.62:36957)
0926 07:27:31.840856 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:31.840886 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:31.842009 (+ 1123us) server_negotiation.cc:407] Connection header received
0926 07:27:31.843572 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:31.843597 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:31.844057 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:31.844408 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:31.846076 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:31.847580 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:31.849283 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:31.850080 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:31.850282 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:31.861499 (+ 11217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:31.861507 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:31.861532 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:31.861574 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:31.863936 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:31.864701 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:31.864708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:31.864714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:31.864867 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:31.865289 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:31.865293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:31.865295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:31.865849 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:31.865985 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:31.868645 (+ 2660us) server_negotiation.cc:299] Negotiation successful
0926 07:27:31.868952 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":107,"threads_started":1}
I20260926 07:27:31.872746 13387 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36957
I20260926 07:27:31.873013 13387 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:31.873572 13387 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
W20260926 07:27:31.875747 13387 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36957 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:27:31.875900 13387 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:31.876068 13387 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
I20260926 07:27:31.876962 13387 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:31.877151 13387 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
W20260926 07:27:31.877955 13387 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:27:31.999768 13391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:32.000140 13391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:32.000276 13391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:32.005398 13391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:32.005457 13391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:32.005569 13391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:27:32.012073 13391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55027/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33369
--tserver_master_addrs=127.0.98.62:36957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:32.013154 13391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:32.014189 13391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:32.023895 13397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:32.023906 13396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:32.023962 13399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:32.024194 13391 server_base.cc:1034] running on GCE node
I20260926 07:27:32.024569 13391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:32.025149 13391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:32.026319 13391 hybrid_clock.cc:648] HybridClock initialized: now 1790407652026296 us; error 46 us; skew 500 ppm
Sep 26 07:27:32 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:32.029131 13391 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:27:32.030054 13391 webserver.cc:460] Webserver started at http://127.0.98.2:33369/ using document root <none> and password file <none>
I20260926 07:27:32.030601 13391 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:32.030671 13391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:27:32 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:27:32.057533 13391 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260926 07:27:32.060251 13407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:32.061674 13391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:32.061818 13391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "5ff850e18da946fe92629e53cd48a05d"
format_stamp: "Formatted at 2026-09-26 07:27:30 on dist-test-slave-g04g"
server_key: "5a03fd444e3f0c4ac46734ef00bebf9b"
server_key_iv: "f421c8a8a746f5a7f10f7a1e80109d06"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:32.062492 13391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:32.075578 13391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:32.079087 13391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:32.079341 13391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:32.080056 13391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:32.081106 13391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:32.081187 13391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:32.081293 13391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:32.081341 13391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:32.092304 13391 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38901
I20260926 07:27:32.092455 13520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38901 every 8 connection(s)
I20260926 07:27:32.093427 13391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:32.096837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 13391
I20260926 07:27:32.097265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55027/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42121
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36957
--webserver_port=35371
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55027/kms with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:27:32 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:32.113490 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:32.095404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53665 (local address 127.0.98.62:36957)
0926 07:27:32.095574 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:32.095578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:32.096443 (+ 865us) server_negotiation.cc:407] Connection header received
0926 07:27:32.100176 (+ 3733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:32.100183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:32.100254 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:32.100316 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:32.102105 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:32.103107 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:32.105080 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:32.105457 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:32.105601 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:32.108975 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:32.108979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:32.108982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:32.109015 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:32.110889 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:32.111557 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:32.111561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:32.111562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:32.111667 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:32.112118 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:32.112122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:32.112124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:32.112359 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:32.112459 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:32.113174 (+ 715us) server_negotiation.cc:299] Negotiation successful
0926 07:27:32.113306 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:27:32.114857 13523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:32.095830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36957 (local address 127.0.98.2:53665)
0926 07:27:32.096293 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:32.096344 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:32.099903 (+ 3559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:32.100761 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:32.100773 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:32.101135 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:32.101903 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:32.101928 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:32.103290 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:32.103298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:32.104935 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:32.104951 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:32.105803 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:32.105807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:32.106114 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:32.106811 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:32.106831 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:32.108762 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:32.111024 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:32.111056 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:32.111073 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:32.111405 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:32.111798 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:32.111801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:32.111807 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:32.111991 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:32.112496 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:32.112503 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:32.112880 (+ 377us) client_negotiation.cc:769] Sending connection context
0926 07:27:32.113102 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 07:27:32.113442 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":138,"threads_started":1}
I20260926 07:27:32.116130 13521 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36957
I20260926 07:27:32.116380 13521 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:32.116930 13521 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
W20260926 07:27:32.118650 13521 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36957 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:27:32.118773 13521 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:32.118917 13521 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
I20260926 07:27:32.119831 13521 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:32.119982 13521 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
W20260926 07:27:32.120801 13521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:27:32.234980 13524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:32.235438 13524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:32.235560 13524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:32.241246 13524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:32.241325 13524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:32.241412 13524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:27:32.247160 13524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55027/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35371
--tserver_master_addrs=127.0.98.62:36957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:32.248234 13524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:32.249279 13524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:32.259322 13531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:32.259552 13524 server_base.cc:1034] running on GCE node
W20260926 07:27:32.259716 13533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:32.259979 13530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:32.260556 13524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:32.261288 13524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:32.262460 13524 hybrid_clock.cc:648] HybridClock initialized: now 1790407652262435 us; error 47 us; skew 500 ppm
Sep 26 07:27:32 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:32.266383 13524 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:27:32.267516 13524 webserver.cc:460] Webserver started at http://127.0.98.3:35371/ using document root <none> and password file <none>
I20260926 07:27:32.268333 13524 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:32.268461 13524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:27:32 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:27:32.290438 13524 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 07:27:32.293836 13541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:32.295452 13524 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 07:27:32.295621 13524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "03f50bee1eb4454c82cced28ab8d5d33"
format_stamp: "Formatted at 2026-09-26 07:27:31 on dist-test-slave-g04g"
server_key: "fdfeed7ad7f716d563a2e1a96c449ca8"
server_key_iv: "c49b1669813e163e41a88571bcf3045a"
server_key_version: "kuduclusterkey@0"
I20260926 07:27:32.296366 13524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:32.327595 13524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:32.331559 13524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:32.331784 13524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:32.332475 13524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:32.333442 13524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:32.333525 13524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:32.333604 13524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:32.333662 13524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:32.347700 13524 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42121
I20260926 07:27:32.347826 13654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42121 every 8 connection(s)
I20260926 07:27:32.348807 13524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:32.349458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 13524
I20260926 07:27:32.363137 13390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:32.350185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.0.98.62:36957)
0926 07:27:32.350328 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:32.350333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:32.351400 (+ 1067us) server_negotiation.cc:407] Connection header received
0926 07:27:32.351595 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:32.351600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:32.351659 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:32.351741 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:32.354995 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:32.355895 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:32.357252 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:32.357523 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:32.357647 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:32.359480 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:32.359485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:32.359488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:32.359521 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:32.361122 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:32.361630 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:32.361637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:32.361639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:32.361727 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:32.362025 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:32.362029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:32.362031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:32.362282 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:32.362514 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:32.362747 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 07:27:32.362872 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 07:27:32 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:32.373724 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:32.353732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36957 (local address 127.0.98.3:40391)
0926 07:27:32.354233 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:32.354289 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:32.355385 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:32.356837 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:32.356847 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:32.357764 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:32.358531 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:32.358560 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:32.361285 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:32.361291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:32.364526 (+ 3235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:32.364538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:32.365278 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:32.365283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:32.365517 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:32.366148 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:32.366165 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:32.367739 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:32.370196 (+ 2457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:32.370203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:32.370220 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:32.370574 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:32.370943 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:32.370946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:32.370948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:32.371208 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:32.371703 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:32.371710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:32.372138 (+ 428us) client_negotiation.cc:769] Sending connection context
0926 07:27:32.372357 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 07:27:32.372694 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":162,"threads_started":1}
I20260926 07:27:32.374498 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:32.354807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40391 (local address 127.0.98.62:36957)
0926 07:27:32.356156 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:32.356160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:32.356172 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:27:32.356206 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:32.356209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:32.356308 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:32.356528 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:32.358719 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:32.360340 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:32.364663 (+ 4323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:32.365113 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:32.365238 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:32.367915 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:32.367919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:32.367922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:32.367946 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:32.370054 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:32.370699 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:32.370703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:32.370705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:32.370797 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:32.371329 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:32.371333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:32.371335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:32.371553 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:32.371648 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:32.374272 (+ 2624us) server_negotiation.cc:299] Negotiation successful
0926 07:27:32.374375 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1272,"thread_start_us":79,"threads_started":1}
I20260926 07:27:32.375557 13655 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36957
I20260926 07:27:32.375841 13655 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:32.376456 13655 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
W20260926 07:27:32.378196 13655 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36957 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:27:32.378317 13655 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:32.378469 13655 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
I20260926 07:27:32.379284 13655 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:32.379443 13655 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
W20260926 07:27:32.380187 13655 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 07:27:32 dist-test-slave-g04g krb5kdc[12193](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790407652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:32.878196 13387 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:32.878491 13387 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
I20260926 07:27:32.879755 13189 ts_manager.cc:194] Registered new tserver with Master: 47b3b25c809b4cab9191efe706735467 (127.0.98.1:46603)
I20260926 07:27:32.881464 13189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47195
Sep 26 07:27:33 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:27:33.121013 13521 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:33.121292 13521 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
I20260926 07:27:33.121984 13189 ts_manager.cc:194] Registered new tserver with Master: 5ff850e18da946fe92629e53cd48a05d (127.0.98.2:38901)
I20260926 07:27:33.122743 13189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53665
Sep 26 07:27:33 dist-test-slave-g04g krb5kdc[12193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 07:27:33.380396 13655 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:33.380715 13655 heartbeater.cc:511] Master 127.0.98.62:36957 requested a full tablet report, sending...
I20260926 07:27:33.381551 13189 ts_manager.cc:194] Registered new tserver with Master: 03f50bee1eb4454c82cced28ab8d5d33 (127.0.98.3:42121)
I20260926 07:27:33.382282 13189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40391
I20260926 07:27:33.390513 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:27:33.413271 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.396559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.0.98.62:36957)
0926 07:27:33.397445 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:33.397450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:33.397862 (+ 412us) server_negotiation.cc:407] Connection header received
0926 07:27:33.398059 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:33.398063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:33.398114 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:33.398174 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:33.399147 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.400004 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.406677 (+ 6673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.407085 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.407170 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.409180 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:33.409184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:33.409187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:33.409218 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:33.411167 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:33.411727 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:33.411731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:33.411733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:33.411820 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:33.412117 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:33.412121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:33.412122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:33.412336 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:33.412440 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:33.412948 (+ 508us) server_negotiation.cc:299] Negotiation successful
0926 07:27:33.413083 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":118,"threads_started":1}
I20260926 07:27:33.418653 13189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:27:33.423691 13189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:33.450064 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.442313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46603 (local address 127.0.0.1:36772)
0926 07:27:33.443914 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:33.443966 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:33.444094 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:33.445229 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:33.445235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:33.445256 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:33.445376 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:33.445382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.446829 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.446832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:33.448237 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:33.448247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.449533 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.449536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:33.449643 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.449680 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:27:33.449788 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 07:27:33.449903 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1420,"thread_start_us":101,"threads_started":1}
I20260926 07:27:33.451097 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.442430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.0.98.1:46603)
0926 07:27:33.444626 (+ 2196us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:33.444633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:33.444654 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:27:33.444726 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:33.444731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:33.444923 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:33.445063 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:33.445523 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.446697 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.448377 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.449223 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.449324 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.449473 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:33.450844 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0926 07:27:33.450943 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2095,"thread_start_us":52,"threads_started":1}
I20260926 07:27:33.454481 13321 tablet_service.cc:1467] Processing CreateTablet for tablet ec00c9c2cc1943919975e903ac0dcceb (DEFAULT_TABLE table=test-table [id=3d0c5a9b1f6f4031b074cfe20af5ec37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:33.455662 13321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec00c9c2cc1943919975e903ac0dcceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:33.463117 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.443460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38901 (local address 127.0.0.1:59546)
0926 07:27:33.444314 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:33.444326 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:33.444397 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:33.454586 (+ 10189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:33.454589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:33.454606 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:33.454698 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:33.454703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.458074 (+ 3371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.458077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:33.459272 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:33.459283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.462723 (+ 3440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.462727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:33.462815 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.462843 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:27:33.462890 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 07:27:33.462955 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":87,"threads_started":1}
I20260926 07:27:33.464038 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.443783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42121 (local address 127.0.0.1:60814)
0926 07:27:33.450656 (+ 6873us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:33.450669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:33.450730 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:33.456256 (+ 5526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:33.456259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:33.456276 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:33.456347 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:33.456352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.460023 (+ 3671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.460027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:33.461176 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:33.461186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.463667 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.463671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:33.463763 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.463803 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 07:27:33.463859 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 07:27:33.463921 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6821,"thread_start_us":83,"threads_started":1}
I20260926 07:27:33.464892 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.443611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.0.98.2:38901)
0926 07:27:33.453546 (+ 9935us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:33.453554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:33.453579 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:27:33.453653 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:33.453658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:33.453856 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:33.454850 (+ 994us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:33.455008 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.456114 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.459420 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.462601 (+ 3181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.463290 (+ 689us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.463431 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:33.463494 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 07:27:33.464752 (+ 1258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9811,"thread_start_us":63,"threads_started":1}
I20260926 07:27:33.465091 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.444017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.0.98.3:42121)
0926 07:27:33.454140 (+ 10123us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:33.454146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:33.454167 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:27:33.454236 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:33.454241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:33.454416 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:33.456492 (+ 2076us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:33.456660 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.457941 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.461323 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.462116 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.464211 (+ 2095us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.464349 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:33.464397 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 07:27:33.464501 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10012,"thread_start_us":110,"threads_started":1}
I20260926 07:27:33.467833 13455 tablet_service.cc:1467] Processing CreateTablet for tablet ec00c9c2cc1943919975e903ac0dcceb (DEFAULT_TABLE table=test-table [id=3d0c5a9b1f6f4031b074cfe20af5ec37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:33.467832 13570 tablet_service.cc:1467] Processing CreateTablet for tablet ec00c9c2cc1943919975e903ac0dcceb (DEFAULT_TABLE table=test-table [id=3d0c5a9b1f6f4031b074cfe20af5ec37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:33.468878 13570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec00c9c2cc1943919975e903ac0dcceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:33.468878 13455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec00c9c2cc1943919975e903ac0dcceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:33.477560 13690 tablet_bootstrap.cc:492] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467: Bootstrap starting.
I20260926 07:27:33.480365 13690 tablet_bootstrap.cc:654] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:33.481222 13690 log.cc:824] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:33.481935 13692 tablet_bootstrap.cc:492] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d: Bootstrap starting.
I20260926 07:27:33.484125 13692 tablet_bootstrap.cc:654] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:33.484920 13692 log.cc:824] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:33.485836 13691 tablet_bootstrap.cc:492] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33: Bootstrap starting.
I20260926 07:27:33.488215 13691 tablet_bootstrap.cc:654] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:33.489068 13691 log.cc:824] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:33.497248 13690 tablet_bootstrap.cc:492] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467: No bootstrap required, opened a new log
I20260926 07:27:33.497529 13690 ts_tablet_manager.cc:1397] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260926 07:27:33.498222 13692 tablet_bootstrap.cc:492] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d: No bootstrap required, opened a new log
I20260926 07:27:33.498502 13692 ts_tablet_manager.cc:1397] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260926 07:27:33.502559 13690 raft_consensus.cc:348] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b3b25c809b4cab9191efe706735467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "5ff850e18da946fe92629e53cd48a05d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38901 } } peers { permanent_uuid: "03f50bee1eb4454c82cced28ab8d5d33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42121 } }
I20260926 07:27:33.502521 13692 raft_consensus.cc:348] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b3b25c809b4cab9191efe706735467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "5ff850e18da946fe92629e53cd48a05d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38901 } } peers { permanent_uuid: "03f50bee1eb4454c82cced28ab8d5d33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42121 } }
I20260926 07:27:33.502841 13690 raft_consensus.cc:374] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:33.502907 13690 raft_consensus.cc:729] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b3b25c809b4cab9191efe706735467, State: Initialized, Role: FOLLOWER
I20260926 07:27:33.502952 13692 raft_consensus.cc:374] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:33.503104 13692 raft_consensus.cc:729] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff850e18da946fe92629e53cd48a05d, State: Initialized, Role: FOLLOWER
I20260926 07:27:33.503180 13691 tablet_bootstrap.cc:492] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33: No bootstrap required, opened a new log
I20260926 07:27:33.503376 13691 ts_tablet_manager.cc:1397] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260926 07:27:33.503491 13690 consensus_queue.cc:260] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b3b25c809b4cab9191efe706735467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "5ff850e18da946fe92629e53cd48a05d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38901 } } peers { permanent_uuid: "03f50bee1eb4454c82cced28ab8d5d33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42121 } }
I20260926 07:27:33.503614 13692 consensus_queue.cc:260] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b3b25c809b4cab9191efe706735467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "5ff850e18da946fe92629e53cd48a05d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38901 } } peers { permanent_uuid: "03f50bee1eb4454c82cced28ab8d5d33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42121 } }
I20260926 07:27:33.504611 13690 ts_tablet_manager.cc:1428] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:27:33.504611 13692 ts_tablet_manager.cc:1428] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260926 07:27:33.504999 13387 heartbeater.cc:503] Master 127.0.98.62:36957 was elected leader, sending a full tablet report...
I20260926 07:27:33.504993 13521 heartbeater.cc:503] Master 127.0.98.62:36957 was elected leader, sending a full tablet report...
I20260926 07:27:33.509110 13691 raft_consensus.cc:348] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b3b25c809b4cab9191efe706735467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "5ff850e18da946fe92629e53cd48a05d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38901 } } peers { permanent_uuid: "03f50bee1eb4454c82cced28ab8d5d33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42121 } }
I20260926 07:27:33.509490 13691 raft_consensus.cc:374] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:33.509603 13691 raft_consensus.cc:729] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f50bee1eb4454c82cced28ab8d5d33, State: Initialized, Role: FOLLOWER
I20260926 07:27:33.510133 13691 consensus_queue.cc:260] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b3b25c809b4cab9191efe706735467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "5ff850e18da946fe92629e53cd48a05d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38901 } } peers { permanent_uuid: "03f50bee1eb4454c82cced28ab8d5d33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42121 } }
I20260926 07:27:33.511278 13691 ts_tablet_manager.cc:1428] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260926 07:27:33.511402 13655 heartbeater.cc:503] Master 127.0.98.62:36957 was elected leader, sending a full tablet report...
I20260926 07:27:33.571363 13698 raft_consensus.cc:482] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:33.571765 13698 raft_consensus.cc:504] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b3b25c809b4cab9191efe706735467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "5ff850e18da946fe92629e53cd48a05d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38901 } } peers { permanent_uuid: "03f50bee1eb4454c82cced28ab8d5d33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42121 } }
I20260926 07:27:33.574946 13698 leader_election.cc:288] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47b3b25c809b4cab9191efe706735467 (127.0.98.1:46603), 5ff850e18da946fe92629e53cd48a05d (127.0.98.2:38901)
I20260926 07:27:33.579914 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.573936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54085 (local address 127.0.98.1:46603)
0926 07:27:33.574067 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:33.574073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:33.575362 (+ 1289us) server_negotiation.cc:407] Connection header received
0926 07:27:33.575583 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:33.575587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:33.575638 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:33.575684 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:33.576086 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.576855 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.578506 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.579173 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.579252 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.579316 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:33.579701 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 07:27:33.579785 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:27:33.580541 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.573536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46603 (local address 127.0.98.3:54085)
0926 07:27:33.575274 (+ 1738us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:33.575289 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:33.575442 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:33.575814 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:33.575817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:33.575832 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:33.575972 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:33.575977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.577010 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.577013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:33.578352 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:33.578363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.579413 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.579424 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:33.579541 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.579567 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:27:33.580288 (+ 721us) client_negotiation.cc:240] Negotiation successful
0926 07:27:33.580368 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1671,"thread_start_us":117,"threads_started":1}
I20260926 07:27:33.581343 13339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec00c9c2cc1943919975e903ac0dcceb" candidate_uuid: "03f50bee1eb4454c82cced28ab8d5d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47b3b25c809b4cab9191efe706735467" is_pre_election: true
I20260926 07:27:33.581682 13339 raft_consensus.cc:2454] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03f50bee1eb4454c82cced28ab8d5d33 in term 0.
I20260926 07:27:33.582248 13545 leader_election.cc:302] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f50bee1eb4454c82cced28ab8d5d33, 47b3b25c809b4cab9191efe706735467; no voters:
I20260926 07:27:33.582649 13698 raft_consensus.cc:2781] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:33.582741 13698 raft_consensus.cc:482] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:33.582790 13698 raft_consensus.cc:3037] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:33.582978 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.574799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42895 (local address 127.0.98.2:38901)
0926 07:27:33.574912 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:33.574917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:33.578665 (+ 3748us) server_negotiation.cc:407] Connection header received
0926 07:27:33.578823 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:33.578826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:33.578881 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:33.578955 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:33.579369 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.580137 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.581666 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.582298 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.582401 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.582461 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:33.582779 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 07:27:33.582863 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:27:33.583391 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.574422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38901 (local address 127.0.98.3:42895)
0926 07:27:33.578541 (+ 4119us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:33.578555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:33.578696 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:33.579093 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:33.579096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:33.579114 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:33.579229 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:33.579237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.580424 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.580427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:33.581527 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:33.581548 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.582531 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.582535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:33.582634 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.582655 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:27:33.583187 (+ 532us) client_negotiation.cc:240] Negotiation successful
0926 07:27:33.583262 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4036,"thread_start_us":103,"threads_started":1}
I20260926 07:27:33.583958 13468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec00c9c2cc1943919975e903ac0dcceb" candidate_uuid: "03f50bee1eb4454c82cced28ab8d5d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff850e18da946fe92629e53cd48a05d" is_pre_election: true
I20260926 07:27:33.584188 13468 raft_consensus.cc:2454] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03f50bee1eb4454c82cced28ab8d5d33 in term 0.
I20260926 07:27:33.584205 13698 raft_consensus.cc:504] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b3b25c809b4cab9191efe706735467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "5ff850e18da946fe92629e53cd48a05d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38901 } } peers { permanent_uuid: "03f50bee1eb4454c82cced28ab8d5d33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42121 } }
I20260926 07:27:33.584597 13698 leader_election.cc:288] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [CANDIDATE]: Term 1 election: Requested vote from peers 47b3b25c809b4cab9191efe706735467 (127.0.98.1:46603), 5ff850e18da946fe92629e53cd48a05d (127.0.98.2:38901)
I20260926 07:27:33.585026 13468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec00c9c2cc1943919975e903ac0dcceb" candidate_uuid: "03f50bee1eb4454c82cced28ab8d5d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff850e18da946fe92629e53cd48a05d"
I20260926 07:27:33.585136 13468 raft_consensus.cc:3037] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:33.585438 13339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec00c9c2cc1943919975e903ac0dcceb" candidate_uuid: "03f50bee1eb4454c82cced28ab8d5d33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47b3b25c809b4cab9191efe706735467"
I20260926 07:27:33.585570 13339 raft_consensus.cc:3037] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:33.586369 13468 raft_consensus.cc:2454] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03f50bee1eb4454c82cced28ab8d5d33 in term 1.
I20260926 07:27:33.586716 13542 leader_election.cc:302] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f50bee1eb4454c82cced28ab8d5d33, 5ff850e18da946fe92629e53cd48a05d; no voters:
I20260926 07:27:33.586930 13339 raft_consensus.cc:2454] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03f50bee1eb4454c82cced28ab8d5d33 in term 1.
I20260926 07:27:33.587221 13698 raft_consensus.cc:2781] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:33.587457 13698 raft_consensus.cc:686] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [term 1 LEADER]: Becoming Leader. State: Replica: 03f50bee1eb4454c82cced28ab8d5d33, State: Running, Role: LEADER
I20260926 07:27:33.587841 13698 consensus_queue.cc:237] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b3b25c809b4cab9191efe706735467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } } peers { permanent_uuid: "5ff850e18da946fe92629e53cd48a05d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38901 } } peers { permanent_uuid: "03f50bee1eb4454c82cced28ab8d5d33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42121 } }
W20260926 07:27:33.591581 13388 tablet.cc:2369] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:33.591640 13188 catalog_manager.cc:5495] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03f50bee1eb4454c82cced28ab8d5d33 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "03f50bee1eb4454c82cced28ab8d5d33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b3b25c809b4cab9191efe706735467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff850e18da946fe92629e53cd48a05d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f50bee1eb4454c82cced28ab8d5d33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42121 } health_report { overall_health: HEALTHY } } }
W20260926 07:27:33.595522 13522 tablet.cc:2369] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:27:33.600863 13656 tablet.cc:2369] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:33.634903 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.630882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.0.98.3:42121)
0926 07:27:33.631013 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:33.631017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:33.631084 (+ 67us) server_negotiation.cc:407] Connection header received
0926 07:27:33.631251 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:33.631253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:33.631307 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:33.631403 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:33.631693 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.632224 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.633434 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:33.633636 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:33.633711 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:33.634059 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:33.634464 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:33.634549 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:33.634674 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 07:27:33.634720 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:27:33.640722 13468 raft_consensus.cc:1261] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d [term 1 FOLLOWER]: Refusing update from remote peer 03f50bee1eb4454c82cced28ab8d5d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:33.641119 13339 raft_consensus.cc:1261] T ec00c9c2cc1943919975e903ac0dcceb P 47b3b25c809b4cab9191efe706735467 [term 1 FOLLOWER]: Refusing update from remote peer 03f50bee1eb4454c82cced28ab8d5d33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:33.641448 13698 consensus_queue.cc:1035] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff850e18da946fe92629e53cd48a05d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:33.641788 13701 consensus_queue.cc:1035] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b3b25c809b4cab9191efe706735467" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:33.649945 13713 mvcc.cc:205] Tried to move back new op lower bound from 7333509749304860672 to 7333509749098864640. Current Snapshot: MvccSnapshot[applied={T|T < 7333509749304860672}]
I20260926 07:27:33.659816 13188 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:33.659988 13188 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:33.662884 13188 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a658dbb5445a4a03a9e84573aa552014: Sending DeleteTablet for 3 replicas of tablet ec00c9c2cc1943919975e903ac0dcceb
I20260926 07:27:33.663733 13321 tablet_service.cc:1514] Processing DeleteTablet for tablet ec00c9c2cc1943919975e903ac0dcceb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36772
I20260926 07:27:33.664244 13455 tablet_service.cc:1514] Processing DeleteTablet for tablet ec00c9c2cc1943919975e903ac0dcceb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59546
I20260926 07:27:33.664476 13570 tablet_service.cc:1514] Processing DeleteTablet for tablet ec00c9c2cc1943919975e903ac0dcceb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60814
I20260926 07:27:33.664830 13722 tablet_replica.cc:330] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d: stopping tablet replica
I20260926 07:27:33.665191 13722 raft_consensus.cc:2229] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:33.665561 13722 raft_consensus.cc:2258] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:33.665669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 13227
I20260926 07:27:33.668311 13722 ts_tablet_manager.cc:1905] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:33.671172 13722 ts_tablet_manager.cc:1918] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:33.671212 13723 tablet_replica.cc:330] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33: stopping tablet replica
I20260926 07:27:33.671344 13722 log.cc:1196] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/ec00c9c2cc1943919975e903ac0dcceb
I20260926 07:27:33.671594 13723 raft_consensus.cc:2229] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:27:33.671774 13722 ts_tablet_manager.cc:1939] T ec00c9c2cc1943919975e903ac0dcceb P 5ff850e18da946fe92629e53cd48a05d: Deleting consensus metadata
I20260926 07:27:33.672092 13723 raft_consensus.cc:2258] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:33.673516 13173 catalog_manager.cc:4841] TS 5ff850e18da946fe92629e53cd48a05d (127.0.98.2:38901): tablet ec00c9c2cc1943919975e903ac0dcceb (table test-table [id=3d0c5a9b1f6f4031b074cfe20af5ec37]) successfully deleted
I20260926 07:27:33.675122 13723 ts_tablet_manager.cc:1905] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:27:33.675999 13176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46603 (error 108)
I20260926 07:27:33.677399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 13391
I20260926 07:27:33.677438 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:33.677145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46603 (local address 127.0.0.1:36778)
0926 07:27:33.677273 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:33.677348 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 07:27:33.677620 13176 catalog_manager.cc:4584] TS 47b3b25c809b4cab9191efe706735467 (127.0.98.1:46603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec00c9c2cc1943919975e903ac0dcceb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46603: connect: Connection refused (error 111)
I20260926 07:27:33.677769 13723 ts_tablet_manager.cc:1918] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:33.677851 13723 log.cc:1196] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/ec00c9c2cc1943919975e903ac0dcceb
I20260926 07:27:33.678197 13723 ts_tablet_manager.cc:1939] T ec00c9c2cc1943919975e903ac0dcceb P 03f50bee1eb4454c82cced28ab8d5d33: Deleting consensus metadata
I20260926 07:27:33.679347 13173 catalog_manager.cc:4841] TS 03f50bee1eb4454c82cced28ab8d5d33 (127.0.98.3:42121): tablet ec00c9c2cc1943919975e903ac0dcceb (table test-table [id=3d0c5a9b1f6f4031b074cfe20af5ec37]) successfully deleted
I20260926 07:27:33.685788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 13524
I20260926 07:27:33.690937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 13156
I20260926 07:27:33.697604 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 07:27:34.032582 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 07:27:34.032716 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 07:27:34.377142 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 07:27:34.377 UTC [12223] LOG: received smart shutdown request
2026-09-26 07:27:34.381 UTC [12223] LOG: background worker "logical replication launcher" (PID 12232) exited with exit code 1
2026-09-26 07:27:34.381 UTC [12227] LOG: shutting down
2026-09-26 07:27:34.548 UTC [12223] LOG: database system is shut down
2026-09-26T07:27:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49871 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:34 dist-test-slave-g04g krb5kdc[13733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:34 dist-test-slave-g04g krb5kdc[13733](info): set up 2 sockets
Sep 26 07:27:34 dist-test-slave-g04g krb5kdc[13733](info): commencing operation
krb5kdc: starting...
W20260926 07:27:35.699210 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:27:35 dist-test-slave-g04g krb5kdc[13733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:27:35.821674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33033
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33033,127.0.98.61:35943,127.0.98.60:41193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:35.932389 13746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:35.932684 13746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:35.932757 13746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:35.936992 13746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:35.937074 13746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:35.937110 13746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:35.937165 13746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:35.937202 13746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:35.942441 13746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33033,127.0.98.61:35943,127.0.98.60:41193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:35.943634 13746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:35.944669 13746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:35.953307 13751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:35.953284 13754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:35.953356 13752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:35.953709 13746 server_base.cc:1034] running on GCE node
I20260926 07:27:35.954265 13746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:35.955397 13746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:35.956558 13746 hybrid_clock.cc:648] HybridClock initialized: now 1790407655956544 us; error 25 us; skew 500 ppm
Sep 26 07:27:35 dist-test-slave-g04g krb5kdc[13733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:35.959456 13746 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:27:35.960316 13746 webserver.cc:460] Webserver started at http://127.0.98.62:43875/ using document root <none> and password file <none>
I20260926 07:27:35.960832 13746 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:35.960887 13746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:35.961045 13746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:35.962965 13746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "eb8834fee04a429a8848c2b1daa5603a"
format_stamp: "Formatted at 2026-09-26 07:27:35 on dist-test-slave-g04g"
server_key: "409b5745a3ab6970784654d3955d14f2"
server_key_iv: "5158d39f4352c43c4295481deef4f774"
server_key_version: "encryptionkey@0"
I20260926 07:27:35.963518 13746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb8834fee04a429a8848c2b1daa5603a"
format_stamp: "Formatted at 2026-09-26 07:27:35 on dist-test-slave-g04g"
server_key: "409b5745a3ab6970784654d3955d14f2"
server_key_iv: "5158d39f4352c43c4295481deef4f774"
server_key_version: "encryptionkey@0"
I20260926 07:27:35.967129 13746 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:35.969403 13761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:35.970634 13746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:35.970767 13746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "eb8834fee04a429a8848c2b1daa5603a"
format_stamp: "Formatted at 2026-09-26 07:27:35 on dist-test-slave-g04g"
server_key: "409b5745a3ab6970784654d3955d14f2"
server_key_iv: "5158d39f4352c43c4295481deef4f774"
server_key_version: "encryptionkey@0"
I20260926 07:27:35.970896 13746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:35.995929 13746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:35.999464 13746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:35.999662 13746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:36.006954 13746 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33033
I20260926 07:27:36.006999 13813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33033 every 8 connection(s)
I20260926 07:27:36.007936 13746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:36.010802 13814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:36.014950 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:27:36.017951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 13746
I20260926 07:27:36.018080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:27:36.018536 392 external_mini_cluster.cc:1428] Setting key 6ab17d6f8981435a526c7ef9bf773ed8
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260926 07:27:36.031147 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.018403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43600 (local address 127.0.98.62:33033)
0926 07:27:36.018682 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.018694 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.018735 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:27:36.019998 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.020003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.020237 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.020467 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.021389 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.022288 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.023362 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.023726 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.023894 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.026232 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.026246 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.026256 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.026276 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.027975 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.028449 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.028455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.028457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.028562 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.028871 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.028876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.028878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.029242 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.029381 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.029780 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.030048 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":176,"thread_start_us":110,"threads_started":1}
I20260926 07:27:36.031144 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.017523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33033 (local address 127.0.0.1:43600)
0926 07:27:36.018174 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.018238 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:36.019652 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:36.020492 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:36.020497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:36.020862 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:36.021217 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.021231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.022425 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.022429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.023235 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.023244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.023877 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.023881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.024029 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.024638 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:36.024654 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:36.026085 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:36.028101 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.028110 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.028119 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.028342 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.028661 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.028664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.028666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.028774 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.029363 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:36.029368 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:36.029527 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 07:27:36.029710 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 07:27:36.030096 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":212,"thread_start_us":129,"threads_started":1}
I20260926 07:27:36.033762 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:27:36.034621 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.034264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35943 (local address 127.0.0.1:43560)
0926 07:27:36.034425 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.034520 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 07:27:36.034895 13765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35943: connect: Connection refused (error 111)
I20260926 07:27:36.035844 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.035590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35943 (local address 127.0.0.1:43568)
0926 07:27:36.035718 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.035772 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:27:36.035992 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35943: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35943: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 07:27:36.046303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35943
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33033,127.0.98.61:35943,127.0.98.60:41193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:27:36.085256 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } attempt: 1
I20260926 07:27:36.086184 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.085880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35943 (local address 127.0.0.1:43582)
0926 07:27:36.086042 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.086107 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 07:27:36.086918 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.086709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35943 (local address 127.0.0.1:43592)
0926 07:27:36.086815 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.086866 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 07:27:36.087062 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35943: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35943: connect: Connection refused (error 111)
I20260926 07:27:36.155228 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } attempt: 2
I20260926 07:27:36.156123 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.155787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35943 (local address 127.0.0.1:43596)
0926 07:27:36.155962 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.156043 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 07:27:36.156965 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.156754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35943 (local address 127.0.0.1:43608)
0926 07:27:36.156858 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.156906 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 07:27:36.157194 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35943: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35943: connect: Connection refused (error 111)
W20260926 07:27:36.160454 13821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:36.160730 13821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:36.160816 13821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:36.164947 13821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:36.165006 13821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:36.165030 13821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:36.165076 13821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:36.165117 13821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:36.171080 13821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33033,127.0.98.61:35943,127.0.98.60:41193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:36.172219 13821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:36.173312 13821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:36.182333 13827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:36.182308 13829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:36.182289 13826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:36.182616 13821 server_base.cc:1034] running on GCE node
I20260926 07:27:36.183329 13821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:36.184502 13821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:36.185689 13821 hybrid_clock.cc:648] HybridClock initialized: now 1790407656185673 us; error 32 us; skew 500 ppm
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:36.188424 13821 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 07:27:36.189301 13821 webserver.cc:460] Webserver started at http://127.0.98.61:43467/ using document root <none> and password file <none>
I20260926 07:27:36.189845 13821 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:36.189957 13821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:36.190197 13821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:36.192236 13821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/data/instance:
uuid: "72b88d9c53d44eba931361e4d46a1aab"
format_stamp: "Formatted at 2026-09-26 07:27:36 on dist-test-slave-g04g"
server_key: "767d96feaddb87c08829318de55f5b0d"
server_key_iv: "609d6e9ed1d6784dee045ef011195869"
server_key_version: "encryptionkey@0"
I20260926 07:27:36.192790 13821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/wal/instance:
uuid: "72b88d9c53d44eba931361e4d46a1aab"
format_stamp: "Formatted at 2026-09-26 07:27:36 on dist-test-slave-g04g"
server_key: "767d96feaddb87c08829318de55f5b0d"
server_key_iv: "609d6e9ed1d6784dee045ef011195869"
server_key_version: "encryptionkey@0"
I20260926 07:27:36.196512 13821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:27:36.198832 13836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:36.200078 13821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:36.200218 13821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/wal
uuid: "72b88d9c53d44eba931361e4d46a1aab"
format_stamp: "Formatted at 2026-09-26 07:27:36 on dist-test-slave-g04g"
server_key: "767d96feaddb87c08829318de55f5b0d"
server_key_iv: "609d6e9ed1d6784dee045ef011195869"
server_key_version: "encryptionkey@0"
I20260926 07:27:36.200326 13821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:36.214287 13821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:36.217796 13821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:36.218016 13821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:36.225204 13821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35943
I20260926 07:27:36.225260 13888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35943 every 8 connection(s)
I20260926 07:27:36.226234 13821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/data/info.pb
I20260926 07:27:36.229044 13889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:36.232580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 13821
I20260926 07:27:36.232697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-1/wal/instance
I20260926 07:27:36.233006 392 external_mini_cluster.cc:1428] Setting key 5c57bcd487f1adeaa2031ba7cf757127
I20260926 07:27:36.233016 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:36.248073 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.235344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.0.98.62:33033)
0926 07:27:36.235485 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.235490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.236541 (+ 1051us) server_negotiation.cc:407] Connection header received
0926 07:27:36.237457 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.237460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.237519 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.237623 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.238761 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.239336 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.240470 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.240761 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.240871 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.244221 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.244228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.244235 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.244270 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.245863 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.246331 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.246338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.246343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.246442 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.246777 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.246783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.246785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.247081 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.247193 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.247680 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.247831 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:27:36.248371 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } attempt: 3
I20260926 07:27:36.248886 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.235878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33033 (local address 127.0.0.1:43608)
0926 07:27:36.236384 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.236427 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:36.237245 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:36.237788 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:36.237795 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:36.238155 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:36.238615 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.238627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.239470 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.239474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.240343 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.240354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.240913 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.240917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.241097 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.241654 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:36.241669 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:36.244053 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:36.245970 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.245975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.245986 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.246229 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.246537 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.246540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.246541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.246666 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.247189 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:36.247195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:36.247429 (+ 234us) client_negotiation.cc:769] Sending connection context
0926 07:27:36.247583 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 07:27:36.247869 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":145,"threads_started":1}
I20260926 07:27:36.250448 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 07:27:36.259270 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.248806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35943 (local address 127.0.0.1:43624)
0926 07:27:36.248937 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.248952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:36.249035 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:36.249735 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:36.249738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:36.249927 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:36.250067 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.250073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.251766 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.251769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.252474 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.252484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.253255 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.253258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.253367 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.253908 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:36.253920 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:36.255319 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:36.257328 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.257334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.257339 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.257675 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.258028 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.258032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.258034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.258133 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.258639 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:36.258643 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:36.258770 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 07:27:36.258918 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 07:27:36.259023 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 07:27:36.259377 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.248936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.0.98.61:35943)
0926 07:27:36.249325 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.249333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.249351 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:27:36.249408 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.249413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.249607 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.249759 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.250184 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.251221 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.252615 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.253129 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.253245 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.255500 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.255511 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.255525 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.255554 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.257218 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.257791 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.257795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.257796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.257911 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.258239 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.258241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.258243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.258523 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.258633 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.259026 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.259201 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":84,"threads_started":1}
I20260926 07:27:36.260242 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.251329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35943 (local address 127.0.0.1:43632)
0926 07:27:36.251434 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.251451 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:36.251532 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:36.251766 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:36.251771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:36.251935 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:36.252074 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.252081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.253075 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.253078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.253667 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.253675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.254361 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.254368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.254464 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.254934 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:36.254946 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:36.256490 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:36.258796 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.258801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.258803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.259071 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.259337 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.259339 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.259341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.259449 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.259815 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:36.259818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:36.259892 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 07:27:36.259998 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 07:27:36.260121 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:27:36.260386 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.251093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43632 (local address 127.0.98.61:35943)
0926 07:27:36.251431 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.251443 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.251500 (+ 57us) server_negotiation.cc:407] Connection header received
0926 07:27:36.251604 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.251607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.251656 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.251751 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.252205 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.252976 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.253802 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.254225 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.254307 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.257109 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.257115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.257117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.257139 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.258676 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.259154 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.259157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.259159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.259239 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.259552 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.259555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.259557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.259733 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.259822 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.260074 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.260194 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":136,"threads_started":1}
I20260926 07:27:36.261575 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:27:36.261665 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 07:27:36.262418 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.262127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50794)
0926 07:27:36.262275 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.262344 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 07:27:36.262842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41193
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33033,127.0.98.61:35943,127.0.98.60:41193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:27:36.263408 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.262905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50798)
0926 07:27:36.263241 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.263308 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249}
W20260926 07:27:36.264230 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41193: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
I20260926 07:27:36.264536 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.262044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50782)
0926 07:27:36.264397 (+ 2353us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.264479 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2254}
W20260926 07:27:36.264745 13840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
I20260926 07:27:36.265712 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.265437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50814)
0926 07:27:36.265570 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.265636 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 07:27:36.265866 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41193: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
I20260926 07:27:36.295419 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } attempt: 1
I20260926 07:27:36.296492 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.296031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50820)
0926 07:27:36.296238 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.296341 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 07:27:36.297314 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.297056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50824)
0926 07:27:36.297179 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.297236 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 07:27:36.297751 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41193: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
I20260926 07:27:36.315146 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } attempt: 1
I20260926 07:27:36.316042 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.315704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50832)
0926 07:27:36.315872 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.315946 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 07:27:36.316811 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.316576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50838)
0926 07:27:36.316695 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.316749 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:27:36.316955 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41193: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
I20260926 07:27:36.372944 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } attempt: 2
I20260926 07:27:36.373870 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.373497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50664)
0926 07:27:36.373677 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.373760 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 07:27:36.374655 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.374414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50680)
0926 07:27:36.374530 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.374586 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:27:36.374837 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41193: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
I20260926 07:27:36.385138 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } attempt: 2
I20260926 07:27:36.385831 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.385584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50694)
0926 07:27:36.385713 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.385765 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 07:27:36.386583 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.386338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50706)
0926 07:27:36.386469 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.386523 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 07:27:36.386813 13889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41193: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41193: connect: Connection refused (error 111)
W20260926 07:27:36.387486 13896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:36.387804 13896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:36.387909 13896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:36.392441 13896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:36.392520 13896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:36.392573 13896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:36.392614 13896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:36.392665 13896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:36.398279 13896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33033,127.0.98.61:35943,127.0.98.60:41193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:36.399362 13896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:36.400345 13896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:36.408576 13903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:36.408576 13902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:36.408573 13905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:36.408905 13896 server_base.cc:1034] running on GCE node
I20260926 07:27:36.409420 13896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:36.410406 13896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:36.411561 13896 hybrid_clock.cc:648] HybridClock initialized: now 1790407656411527 us; error 54 us; skew 500 ppm
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:36.414247 13896 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 07:27:36.415195 13896 webserver.cc:460] Webserver started at http://127.0.98.60:39581/ using document root <none> and password file <none>
I20260926 07:27:36.415745 13896 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:36.415832 13896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:36.416038 13896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:36.418048 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/data/instance:
uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb"
format_stamp: "Formatted at 2026-09-26 07:27:36 on dist-test-slave-g04g"
server_key: "fda4a5d5c67813532ad7a9d12a4f681a"
server_key_iv: "c36db36842d29678fc8f6c05d29ebda3"
server_key_version: "encryptionkey@0"
I20260926 07:27:36.418648 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/wal/instance:
uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb"
format_stamp: "Formatted at 2026-09-26 07:27:36 on dist-test-slave-g04g"
server_key: "fda4a5d5c67813532ad7a9d12a4f681a"
server_key_iv: "c36db36842d29678fc8f6c05d29ebda3"
server_key_version: "encryptionkey@0"
I20260926 07:27:36.422168 13896 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:27:36.424526 13912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:36.425726 13896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:27:36.425860 13896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/wal
uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb"
format_stamp: "Formatted at 2026-09-26 07:27:36 on dist-test-slave-g04g"
server_key: "fda4a5d5c67813532ad7a9d12a4f681a"
server_key_iv: "c36db36842d29678fc8f6c05d29ebda3"
server_key_version: "encryptionkey@0"
I20260926 07:27:36.425966 13896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:36.439648 13896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:36.442533 13896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:36.442747 13896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:36.449730 13896 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41193
I20260926 07:27:36.449757 13964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41193 every 8 connection(s)
I20260926 07:27:36.450666 13896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/data/info.pb
I20260926 07:27:36.453284 13965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:36.456943 13965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:27:36.460110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 13896
I20260926 07:27:36.460243 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790407489657243-392-0/minicluster-data/master-2/wal/instance
I20260926 07:27:36.460513 392 external_mini_cluster.cc:1428] Setting key d78e8fffec52397900fd83fb00654230
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 07:27:36.470443 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.458763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.0.98.62:33033)
0926 07:27:36.458913 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.458918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.459770 (+ 852us) server_negotiation.cc:407] Connection header received
0926 07:27:36.460766 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.460770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.460828 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.460926 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.462202 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.462715 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.463823 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.464041 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.464154 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.466767 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.466770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.466772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.466794 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.468267 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.468802 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.468807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.468809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.468944 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.469329 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.469333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.469335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.469561 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.469671 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.470156 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.470262 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:27:36.471345 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.459159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33033 (local address 127.0.0.1:44266)
0926 07:27:36.459589 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.459630 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:36.460498 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:36.461128 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:36.461137 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:36.461573 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:36.462041 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.462054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.462844 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.462848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.463692 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.463701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.464194 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.464198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.464359 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.464931 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:36.464946 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:36.466588 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:36.468381 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.468391 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.468400 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.468698 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.469035 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.469041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.469044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.469224 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.469673 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:36.469680 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:36.469897 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 07:27:36.470102 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 07:27:36.470419 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260926 07:27:36.471846 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.461187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.0.98.60:41193)
0926 07:27:36.461441 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.461454 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.461674 (+ 220us) server_negotiation.cc:407] Connection header received
0926 07:27:36.461807 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.461813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.461967 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.462126 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.462847 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.463678 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.464502 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.465015 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.465139 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.468071 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.468077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.468085 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.468105 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.469904 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.470360 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.470367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.470369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.470447 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.470782 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.470785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.470787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.470998 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.471270 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.471492 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.471624 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260926 07:27:36.472946 13965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:27:36.474030 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } attempt: 3
I20260926 07:27:36.477974 13889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } attempt: 3
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:36.483691 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.473348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35943 (local address 127.0.0.1:58202)
0926 07:27:36.473451 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.473465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:36.473528 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:36.473911 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:36.473914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:36.474140 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:36.474290 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.474301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.475342 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.475347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.476276 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.476285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.476895 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.476898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.476993 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.477446 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:36.477461 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:36.478956 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:36.481001 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.481004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.481006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.481360 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.481692 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.481697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.481703 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.481799 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.483224 (+ 1425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:36.483227 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:36.483303 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 07:27:36.483422 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 07:27:36.483537 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":51}
I20260926 07:27:36.483721 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.473481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.0.98.61:35943)
0926 07:27:36.473635 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.473639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.473653 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:27:36.473695 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.473698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.473783 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.473865 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.474437 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.475251 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.476409 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.476779 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.476889 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.479141 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.479144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.479147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.479173 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.480873 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.481482 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.481486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.481488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.481572 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.481902 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.481905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.481907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.482370 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.482460 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.483473 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.483585 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:27:36.483866 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.474392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50722)
0926 07:27:36.474550 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.474563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:36.474641 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:36.474889 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:36.474892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:36.475101 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:36.475185 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.475191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.476046 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.476050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.476763 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.476774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.477646 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.477649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.477751 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.478241 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:36.478255 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:36.479990 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:36.481905 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.481908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.481911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.482127 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.482723 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.482728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.482730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.482835 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.483295 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:36.483301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:36.483408 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 07:27:36.483515 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:27:36.483681 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 07:27:36.484357 13965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:27:36.484844 13968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.474441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.0.98.60:41193)
0926 07:27:36.474547 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.474550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.474606 (+ 56us) server_negotiation.cc:407] Connection header received
0926 07:27:36.474741 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.474744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.474784 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.474859 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.475334 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.475933 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.477187 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.477507 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.477955 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.480159 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.480163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.480165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.480188 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.481794 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.482531 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.482534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.482536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.482611 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.482936 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.482939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.482941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.483154 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.483232 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.484565 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.484676 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":36}
I20260926 07:27:36.486162 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.474074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44280 (local address 127.0.98.62:33033)
0926 07:27:36.474176 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.474182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.474386 (+ 204us) server_negotiation.cc:407] Connection header received
0926 07:27:36.474565 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.474567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.474609 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.474658 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.475537 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.476282 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.477137 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.477504 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.477623 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.482097 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.482101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.482103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.482125 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.483861 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.484837 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.484840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.484842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.484925 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.485232 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.485236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.485237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.485451 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.485659 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.485919 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.486031 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:27:36.488186 13814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a: Bootstrap starting.
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 07:27:36.491084 13814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:36.491994 13814 log.cc:824] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:36.492777 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.478494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50726)
0926 07:27:36.478669 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.478684 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:36.478758 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:36.479245 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:36.479248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:36.479480 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:36.479598 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.479609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.482333 (+ 2724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.482336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.483086 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.483096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.485290 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.485294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.485372 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.487367 (+ 1995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:36.487382 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:36.488823 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:36.491000 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.491004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.491006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.491299 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.491645 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.491650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.491653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.491776 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.492245 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:36.492248 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:36.492345 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 07:27:36.492452 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:27:36.492594 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"spinlock_wait_cycles":14592}
I20260926 07:27:36.492873 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.478548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.0.98.60:41193)
0926 07:27:36.478987 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.478996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.479010 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:27:36.479076 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.479078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.479122 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.479226 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.479994 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.482220 (+ 2226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.484377 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.484710 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.485291 (+ 581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.488994 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.489001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.489009 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.489043 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.490819 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.491448 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.491452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.491455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.491529 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.491901 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.491907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.491912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.492131 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.492254 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.492538 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.492668 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":99,"threads_started":1}
Sep 26 07:27:36 dist-test-slave-g04g krb5kdc[13733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 07:27:36.495111 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.484812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41193 (local address 127.0.0.1:50732)
0926 07:27:36.485064 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:36.485078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:36.485146 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:36.485952 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:36.485954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:36.486120 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:36.486198 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.486205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.487136 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.487140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.487973 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:36.487989 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.488648 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.488651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:36.488746 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.489172 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:36.489188 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:36.491183 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:36.493406 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.493409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.493412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.493683 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.494040 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:36.494042 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:36.494044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:36.494122 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:36.494551 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:36.494556 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:36.494626 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 07:27:36.494733 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:27:36.494917 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260926 07:27:36.495182 13814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a: No bootstrap required, opened a new log
I20260926 07:27:36.495743 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab: Bootstrap starting.
I20260926 07:27:36.496011 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.484838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.0.98.60:41193)
0926 07:27:36.485121 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.485129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.485140 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:27:36.485711 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.485714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.485754 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.485840 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.486325 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.487006 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.488100 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.488454 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.488550 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.491313 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.491318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.491324 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.491352 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.493294 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.493820 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.493826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.493829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.493915 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.494231 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.494233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.494235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.494419 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.494513 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.495412 (+ 899us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.495521 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":138,"threads_started":1}
I20260926 07:27:36.498284 13889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:36.499225 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.488210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.0.98.61:35943)
0926 07:27:36.488319 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.488323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.488465 (+ 142us) server_negotiation.cc:407] Connection header received
0926 07:27:36.488621 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.488624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.488661 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.488706 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.489486 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.490190 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.491260 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.491565 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.491637 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.495355 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.495358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.495361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.495384 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.497091 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.497744 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.497747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.497749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.497823 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.498121 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.498125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.498126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.498345 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.498497 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.498906 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.499014 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:27:36.499198 13889 log.cc:824] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:36.499338 13965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb: Bootstrap starting.
I20260926 07:27:36.499243 13814 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } }
I20260926 07:27:36.499449 13814 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:36.499517 13814 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8834fee04a429a8848c2b1daa5603a, State: Initialized, Role: FOLLOWER
I20260926 07:27:36.500042 13814 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } }
I20260926 07:27:36.501225 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab: No bootstrap required, opened a new log
I20260926 07:27:36.501340 13814 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:27:36.501384 13767 tablet.cc:2369] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:36.501967 13965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:36.502775 13965 log.cc:824] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:36.503834 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:27:36.505416 13889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } }
I20260926 07:27:36.505615 13889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:36.505676 13889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b88d9c53d44eba931361e4d46a1aab, State: Initialized, Role: FOLLOWER
I20260926 07:27:36.506115 13889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } }
W20260926 07:27:36.506697 13986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:27:36.506773 13986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:27:36.507359 13889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:36.507326 13746 master_runner.cc:186] Error getting master registration for 127.0.98.61:35943: OK, instance_id { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" instance_seqno: 1790407656223222 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:27:36.510980 13746 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 07:27:36.512511 14006 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:27:36.512650 14006 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:27:36.513818 13992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } } }
I20260926 07:27:36.513931 13992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:27:36.515298 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } } }
I20260926 07:27:36.515424 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:27:36.515548 13965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb: No bootstrap required, opened a new log
I20260926 07:27:36.514422 13821 master_runner.cc:186] Error getting master registration for 127.0.98.60:41193: OK, instance_id { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" instance_seqno: 1790407656447783 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:27:36.517686 13821 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:27:36.517973 13974 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:36.518111 13974 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } }
I20260926 07:27:36.519168 13974 leader_election.cc:288] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72b88d9c53d44eba931361e4d46a1aab (127.0.98.61:35943), c66bdaf3fe1b4926894dd2dd7a0dbecb (127.0.98.60:41193)
I20260926 07:27:36.519770 13862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb8834fee04a429a8848c2b1daa5603a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b88d9c53d44eba931361e4d46a1aab" is_pre_election: true
I20260926 07:27:36.520031 13862 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb8834fee04a429a8848c2b1daa5603a in term 0.
I20260926 07:27:36.520567 13765 leader_election.cc:302] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72b88d9c53d44eba931361e4d46a1aab, eb8834fee04a429a8848c2b1daa5603a; no voters:
I20260926 07:27:36.520814 13974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:36.520884 13974 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:36.520928 13974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:36.520237 13965 raft_consensus.cc:348] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } }
I20260926 07:27:36.521085 13965 raft_consensus.cc:374] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:36.521165 13965 raft_consensus.cc:729] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66bdaf3fe1b4926894dd2dd7a0dbecb, State: Initialized, Role: FOLLOWER
I20260926 07:27:36.522187 13965 consensus_queue.cc:260] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } }
I20260926 07:27:36.522732 13974 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } }
I20260926 07:27:36.523402 13862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb8834fee04a429a8848c2b1daa5603a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b88d9c53d44eba931361e4d46a1aab"
I20260926 07:27:36.523536 13862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:36.524366 13965 sys_catalog.cc:564] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:36.524456 13939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb8834fee04a429a8848c2b1daa5603a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" is_pre_election: true
I20260926 07:27:36.524705 13939 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb8834fee04a429a8848c2b1daa5603a in term 0.
I20260926 07:27:36.524863 13862 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb8834fee04a429a8848c2b1daa5603a in term 1.
I20260926 07:27:36.525115 13765 leader_election.cc:302] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72b88d9c53d44eba931361e4d46a1aab, eb8834fee04a429a8848c2b1daa5603a; no voters:
I20260926 07:27:36.525283 13974 leader_election.cc:288] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [CANDIDATE]: Term 1 election: Requested vote from peers 72b88d9c53d44eba931361e4d46a1aab (127.0.98.61:35943), c66bdaf3fe1b4926894dd2dd7a0dbecb (127.0.98.60:41193)
I20260926 07:27:36.525377 13974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:36.525453 13974 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [term 1 LEADER]: Becoming Leader. State: Replica: eb8834fee04a429a8848c2b1daa5603a, State: Running, Role: LEADER
I20260926 07:27:36.525728 13974 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } }
I20260926 07:27:36.526012 13939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb8834fee04a429a8848c2b1daa5603a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb"
I20260926 07:27:36.526134 13939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:36.527256 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.506995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.0.98.61:35943)
0926 07:27:36.507150 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.507154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.517937 (+ 10783us) server_negotiation.cc:407] Connection header received
0926 07:27:36.518111 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.518114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.518159 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.518208 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.519267 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.519964 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.521075 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.521362 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.521455 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.523423 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.523426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.523429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.523453 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.525372 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.525837 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.525841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.525843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.525920 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.526240 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.526244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.526246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.526458 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.526581 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.526964 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.527101 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:27:36.527482 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb8834fee04a429a8848c2b1daa5603a. Latest consensus state: current_term: 1 leader_uuid: "eb8834fee04a429a8848c2b1daa5603a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } } }
I20260926 07:27:36.527542 13939 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb8834fee04a429a8848c2b1daa5603a in term 1.
I20260926 07:27:36.527592 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:36.528903 14021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:36.529899 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.506305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.0.98.62:33033)
0926 07:27:36.506429 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.506433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.515692 (+ 9259us) server_negotiation.cc:407] Connection header received
0926 07:27:36.515829 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.515831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.515882 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.515959 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.516975 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.517753 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.519405 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.519763 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.519837 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.522108 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.522113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.522117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.522148 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.523831 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.524246 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.524250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.524251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.524334 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.528406 (+ 4072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.528410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.528412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.528624 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.528748 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.529616 (+ 868us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.529730 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:27:36.530005 14021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:36.524151 14008 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } } }
I20260926 07:27:36.531947 14008 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:27:36.540128 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:36.507674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.0.98.60:41193)
0926 07:27:36.511151 (+ 3477us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:36.511155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:36.527993 (+ 16838us) server_negotiation.cc:407] Connection header received
0926 07:27:36.528159 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:36.528174 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:36.528220 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:36.528296 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:36.531856 (+ 3560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.532487 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.533603 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:36.533967 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:36.534044 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:36.535845 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:36.535849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:36.535852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:36.535877 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:36.537954 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.538457 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.538461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.538463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.538552 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:36.538859 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:36.538863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:36.538864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:36.539119 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:36.539270 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:36.539883 (+ 613us) server_negotiation.cc:299] Negotiation successful
0926 07:27:36.539992 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3386}
W20260926 07:27:36.541424 14022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:27:36.541553 14022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:27:36.551307 13939 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [term 1 FOLLOWER]: Refusing update from remote peer eb8834fee04a429a8848c2b1daa5603a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:36.551412 13862 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [term 1 FOLLOWER]: Refusing update from remote peer eb8834fee04a429a8848c2b1daa5603a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:36.552172 13974 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:36.552583 13974 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:36.557969 13992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb8834fee04a429a8848c2b1daa5603a. Latest consensus state: current_term: 1 leader_uuid: "eb8834fee04a429a8848c2b1daa5603a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } } }
I20260926 07:27:36.558086 13992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:27:36.561105 14008 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb8834fee04a429a8848c2b1daa5603a. Latest consensus state: current_term: 1 leader_uuid: "eb8834fee04a429a8848c2b1daa5603a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } } }
I20260926 07:27:36.561256 14008 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:27:36.574018 14010 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb8834fee04a429a8848c2b1daa5603a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } } }
I20260926 07:27:36.574177 14010 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:36.575573 13992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb8834fee04a429a8848c2b1daa5603a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } } }
I20260926 07:27:36.575688 13992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:27:36.578002 14023 mvcc.cc:205] Tried to move back new op lower bound from 7333509761206165504 to 7333509761132576768. Current Snapshot: MvccSnapshot[applied={T|T < 7333509761206165504}]
I20260926 07:27:36.578886 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb8834fee04a429a8848c2b1daa5603a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } } }
I20260926 07:27:36.578997 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:36.580000 14008 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb8834fee04a429a8848c2b1daa5603a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb8834fee04a429a8848c2b1daa5603a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33033 } } peers { permanent_uuid: "72b88d9c53d44eba931361e4d46a1aab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35943 } } peers { permanent_uuid: "c66bdaf3fe1b4926894dd2dd7a0dbecb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41193 } } }
I20260926 07:27:36.580119 14008 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:27:36.580713 14021 catalog_manager.cc:1347] Generated new cluster ID: d5b9666cf9c149fa9e2c0b54d4b4ed8f
I20260926 07:27:36.580778 14021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:36.580766 14027 mvcc.cc:205] Tried to move back new op lower bound from 7333509761206165504 to 7333509761132576768. Current Snapshot: MvccSnapshot[applied={T|T < 7333509761206165504}]
I20260926 07:27:36.601008 14021 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:36.602226 14021 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:36.622253 14021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb8834fee04a429a8848c2b1daa5603a: Generated new TSK 0
I20260926 07:27:36.623276 14021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:27:37.513908 14006 catalog_manager.cc:1259] Loaded cluster ID: d5b9666cf9c149fa9e2c0b54d4b4ed8f
I20260926 07:27:37.513996 14006 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab: loading cluster ID for follower catalog manager: success
I20260926 07:27:37.515749 14006 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab: acquiring CA information for follower catalog manager: success
I20260926 07:27:37.516947 14006 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 72b88d9c53d44eba931361e4d46a1aab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:27:37.542920 14022 catalog_manager.cc:1259] Loaded cluster ID: d5b9666cf9c149fa9e2c0b54d4b4ed8f
I20260926 07:27:37.543012 14022 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb: loading cluster ID for follower catalog manager: success
I20260926 07:27:37.544750 14022 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb: acquiring CA information for follower catalog manager: success
I20260926 07:27:37.545902 14022 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c66bdaf3fe1b4926894dd2dd7a0dbecb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:27:37.705840 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 13746
I20260926 07:27:37.713130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 13821
I20260926 07:27:37.719424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 13896
2026-09-26T07:27:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3057 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:37 dist-test-slave-g04g krb5kdc[14039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:37 dist-test-slave-g04g krb5kdc[14039](info): set up 2 sockets
Sep 26 07:27:37 dist-test-slave-g04g krb5kdc[14039](info): commencing operation
krb5kdc: starting...
W20260926 07:27:38.755272 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:27:38 dist-test-slave-g04g krb5kdc[14039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:27:38.877950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36179
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:38.986526 14052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:38.986801 14052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:38.986912 14052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:38.990861 14052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:38.990947 14052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:38.990985 14052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:38.991022 14052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:38.991077 14052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:38.996268 14052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:38.997246 14052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:38.998234 14052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:39.006652 14060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:39.006836 14057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:39.006795 14052 server_base.cc:1034] running on GCE node
W20260926 07:27:39.006970 14058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:39.007515 14052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:39.008587 14052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:39.009771 14052 hybrid_clock.cc:648] HybridClock initialized: now 1790407659009760 us; error 27 us; skew 500 ppm
Sep 26 07:27:39 dist-test-slave-g04g krb5kdc[14039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:39.012512 14052 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:27:39.013413 14052 webserver.cc:460] Webserver started at http://127.0.98.62:43081/ using document root <none> and password file <none>
I20260926 07:27:39.013952 14052 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:39.014021 14052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:39.014186 14052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:39.016122 14052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "5371419f21ef45c98fc12e0bce629503"
format_stamp: "Formatted at 2026-09-26 07:27:39 on dist-test-slave-g04g"
server_key: "9e2b1df6ea2348b0d0b0ab6988f5271f"
server_key_iv: "a6062c7b46b2c56257ae80368bd1c7a9"
server_key_version: "encryptionkey@0"
I20260926 07:27:39.016637 14052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "5371419f21ef45c98fc12e0bce629503"
format_stamp: "Formatted at 2026-09-26 07:27:39 on dist-test-slave-g04g"
server_key: "9e2b1df6ea2348b0d0b0ab6988f5271f"
server_key_iv: "a6062c7b46b2c56257ae80368bd1c7a9"
server_key_version: "encryptionkey@0"
I20260926 07:27:39.020151 14052 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:27:39.022354 14067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:39.023599 14052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 07:27:39.023718 14052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "5371419f21ef45c98fc12e0bce629503"
format_stamp: "Formatted at 2026-09-26 07:27:39 on dist-test-slave-g04g"
server_key: "9e2b1df6ea2348b0d0b0ab6988f5271f"
server_key_iv: "a6062c7b46b2c56257ae80368bd1c7a9"
server_key_version: "encryptionkey@0"
I20260926 07:27:39.023823 14052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:39.038012 14052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:39.041188 14052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:39.041385 14052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:39.048369 14052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36179
I20260926 07:27:39.048396 14119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36179 every 8 connection(s)
I20260926 07:27:39.049266 14052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:39.051955 14120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:39.053686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 14052
I20260926 07:27:39.053799 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:27:39.054095 392 external_mini_cluster.cc:1428] Setting key b40137dcc009629afa9a8143a2df0d35
I20260926 07:27:39.058123 14120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503: Bootstrap starting.
I20260926 07:27:39.060346 14120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:39.061064 14120 log.cc:824] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:27:39 dist-test-slave-g04g krb5kdc[14039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:39.062887 14120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503: No bootstrap required, opened a new log
I20260926 07:27:39.066154 14120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5371419f21ef45c98fc12e0bce629503" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36179 } }
I20260926 07:27:39.066383 14120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:39.066481 14120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5371419f21ef45c98fc12e0bce629503, State: Initialized, Role: FOLLOWER
I20260926 07:27:39.066918 14120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5371419f21ef45c98fc12e0bce629503" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36179 } }
I20260926 07:27:39.067154 14120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:39.067233 14120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:39.067330 14120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:39.067620 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:39.055423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.0.98.62:36179)
0926 07:27:39.055843 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:39.055858 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:39.055893 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:27:39.056522 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:39.056540 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:39.056854 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:39.057180 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:39.057941 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:39.058972 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:39.059867 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:39.060332 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:39.060505 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:39.062918 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:39.062923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:39.062935 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:39.062960 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:39.064681 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:39.065108 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:39.065114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:39.065120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:39.065231 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:39.065510 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:39.065515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:39.065516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:39.065882 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:39.066094 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:39.066349 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 07:27:39.066663 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":117,"threads_started":1}
I20260926 07:27:39.068228 14120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5371419f21ef45c98fc12e0bce629503" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36179 } }
I20260926 07:27:39.068593 14120 leader_election.cc:302] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5371419f21ef45c98fc12e0bce629503; no voters:
I20260926 07:27:39.068909 14120 leader_election.cc:288] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:39.069046 14125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:39.069269 14125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [term 1 LEADER]: Becoming Leader. State: Replica: 5371419f21ef45c98fc12e0bce629503, State: Running, Role: LEADER
I20260926 07:27:39.069612 14125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5371419f21ef45c98fc12e0bce629503" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36179 } }
I20260926 07:27:39.070163 14120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:39.070654 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:27:39.071185 14127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5371419f21ef45c98fc12e0bce629503. Latest consensus state: current_term: 1 leader_uuid: "5371419f21ef45c98fc12e0bce629503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5371419f21ef45c98fc12e0bce629503" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36179 } } }
I20260926 07:27:39.071306 14127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:39.071720 14134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:39.072573 14126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5371419f21ef45c98fc12e0bce629503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5371419f21ef45c98fc12e0bce629503" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36179 } } }
I20260926 07:27:39.072669 14126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:39.074709 14134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:39.079574 14134 catalog_manager.cc:1347] Generated new cluster ID: 0ddcb296499147b3a7cb84f4e028f79e
I20260926 07:27:39.079639 14134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:39.116331 14134 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:39.117648 14134 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:39.127306 14134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5371419f21ef45c98fc12e0bce629503: Generated new TSK 0
I20260926 07:27:39.128111 14134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:27:39.177807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 14052
2026-09-26T07:27:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1456 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:39 dist-test-slave-g04g krb5kdc[14147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:39 dist-test-slave-g04g krb5kdc[14147](info): set up 2 sockets
Sep 26 07:27:39 dist-test-slave-g04g krb5kdc[14147](info): commencing operation
krb5kdc: starting...
W20260926 07:27:40.263453 392 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:27:40 dist-test-slave-g04g krb5kdc[14147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:27:40.385850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42159
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:40.494999 14160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:40.495312 14160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:40.495393 14160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:40.499372 14160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:40.499456 14160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:40.499500 14160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:40.499557 14160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:40.499595 14160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:40.504866 14160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:40.505955 14160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:40.506965 14160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:40.515443 14168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:40.515362 14166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:40.515393 14165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:40.515573 14160 server_base.cc:1034] running on GCE node
I20260926 07:27:40.516328 14160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:40.517464 14160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:40.518643 14160 hybrid_clock.cc:648] HybridClock initialized: now 1790407660518621 us; error 50 us; skew 500 ppm
Sep 26 07:27:40 dist-test-slave-g04g krb5kdc[14147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:40.521420 14160 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:27:40.522255 14160 webserver.cc:460] Webserver started at http://127.0.98.62:37935/ using document root <none> and password file <none>
I20260926 07:27:40.522810 14160 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:40.522900 14160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:40.523175 14160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:40.525168 14160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "eebffc3380404166a99eb1ae392d0814"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "5fcae58da9f843d3cb844e709e86df49"
server_key_iv: "cf565fce1eea765bb9b83c0726e2498d"
server_key_version: "encryptionkey@0"
I20260926 07:27:40.525733 14160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "eebffc3380404166a99eb1ae392d0814"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "5fcae58da9f843d3cb844e709e86df49"
server_key_iv: "cf565fce1eea765bb9b83c0726e2498d"
server_key_version: "encryptionkey@0"
I20260926 07:27:40.529382 14160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260926 07:27:40.531795 14175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:40.532979 14160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260926 07:27:40.533115 14160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "eebffc3380404166a99eb1ae392d0814"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "5fcae58da9f843d3cb844e709e86df49"
server_key_iv: "cf565fce1eea765bb9b83c0726e2498d"
server_key_version: "encryptionkey@0"
I20260926 07:27:40.533222 14160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:40.557217 14160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:40.560117 14160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:40.560317 14160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:40.566962 14160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42159
I20260926 07:27:40.567266 14227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42159 every 8 connection(s)
I20260926 07:27:40.567945 14160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:40.570673 14228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:40.572011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 14160
I20260926 07:27:40.572158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:27:40.572454 392 external_mini_cluster.cc:1428] Setting key 75e0cfa783d269f9e1ae645ab4acf563
I20260926 07:27:40.576848 14228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Bootstrap starting.
I20260926 07:27:40.579396 14228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:40.580287 14228 log.cc:824] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:27:40 dist-test-slave-g04g krb5kdc[14147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:40.582176 14228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: No bootstrap required, opened a new log
I20260926 07:27:40.585475 14228 raft_consensus.cc:348] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } }
I20260926 07:27:40.585793 14228 raft_consensus.cc:374] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:40.585873 14228 raft_consensus.cc:729] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebffc3380404166a99eb1ae392d0814, State: Initialized, Role: FOLLOWER
I20260926 07:27:40.586297 14228 consensus_queue.cc:260] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } }
I20260926 07:27:40.586448 14228 raft_consensus.cc:388] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:40.586534 14228 raft_consensus.cc:482] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:40.586628 14228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:40.586654 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:40.573788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.0.98.62:42159)
0926 07:27:40.574163 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:40.574190 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:40.574245 (+ 55us) server_negotiation.cc:407] Connection header received
0926 07:27:40.574869 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:40.574891 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:40.575280 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:40.575560 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:40.576470 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:40.577442 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:40.578591 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:40.579085 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:40.579208 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:40.581906 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:40.581911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:40.581924 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:40.581950 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:40.583728 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:40.584170 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:40.584181 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:40.584185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:40.584289 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:40.584572 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:40.584576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:40.584578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:40.584951 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:40.585157 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:40.585453 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 07:27:40.585726 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":120,"threads_started":1}
I20260926 07:27:40.587730 14228 raft_consensus.cc:504] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } }
I20260926 07:27:40.588071 14228 leader_election.cc:302] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eebffc3380404166a99eb1ae392d0814; no voters:
I20260926 07:27:40.588362 14228 leader_election.cc:288] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:40.588465 14233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:40.588663 14233 raft_consensus.cc:686] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 1 LEADER]: Becoming Leader. State: Replica: eebffc3380404166a99eb1ae392d0814, State: Running, Role: LEADER
I20260926 07:27:40.589025 14233 consensus_queue.cc:237] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } }
I20260926 07:27:40.589583 14228 sys_catalog.cc:564] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:40.590584 14235 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eebffc3380404166a99eb1ae392d0814. Latest consensus state: current_term: 1 leader_uuid: "eebffc3380404166a99eb1ae392d0814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } } }
I20260926 07:27:40.590704 14235 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:40.591115 14242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:40.591097 14234 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eebffc3380404166a99eb1ae392d0814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } } }
I20260926 07:27:40.591769 14234 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:40.594122 14242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:40.599620 14242 catalog_manager.cc:1347] Generated new cluster ID: d65d00b38d01403f8b9fccb639daeb4f
I20260926 07:27:40.599694 14242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:40.612116 14242 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:40.613081 14242 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:40.619694 14242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Generated new TSK 0
I20260926 07:27:40.620596 14242 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:27:40.659305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42159
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:40.767663 14254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:40.767896 14254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:40.767937 14254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:40.771927 14254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:40.771981 14254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:40.772037 14254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:27:40.777094 14254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:40.778124 14254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:40.779179 14254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:40.789083 14259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:40.789132 14262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:40.789083 14260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:40.789321 14254 server_base.cc:1034] running on GCE node
I20260926 07:27:40.789899 14254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:40.790647 14254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:40.791798 14254 hybrid_clock.cc:648] HybridClock initialized: now 1790407660791770 us; error 70 us; skew 500 ppm
Sep 26 07:27:40 dist-test-slave-g04g krb5kdc[14147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:40.794580 14254 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:27:40.795542 14254 webserver.cc:460] Webserver started at http://127.0.98.1:33165/ using document root <none> and password file <none>
I20260926 07:27:40.796115 14254 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:40.796175 14254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:40.796391 14254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:40.798413 14254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "d0245ba57cdb486a9d045eff69a77388"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "62a3d9a97e62eec2cf0ef41afd629ad7"
server_key_iv: "2ef8c00b34bfac0ea7354dc63c82db55"
server_key_version: "encryptionkey@0"
I20260926 07:27:40.798961 14254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d0245ba57cdb486a9d045eff69a77388"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "62a3d9a97e62eec2cf0ef41afd629ad7"
server_key_iv: "2ef8c00b34bfac0ea7354dc63c82db55"
server_key_version: "encryptionkey@0"
I20260926 07:27:40.802579 14254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:40.805116 14269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:40.806382 14254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:40.806514 14254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "d0245ba57cdb486a9d045eff69a77388"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "62a3d9a97e62eec2cf0ef41afd629ad7"
server_key_iv: "2ef8c00b34bfac0ea7354dc63c82db55"
server_key_version: "encryptionkey@0"
I20260926 07:27:40.806615 14254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:40.820060 14254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:40.823340 14254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:40.823537 14254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:40.824103 14254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:40.824884 14254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:40.824929 14254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:40.825003 14254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:40.825035 14254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:40.833727 14254 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32917
I20260926 07:27:40.833752 14382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32917 every 8 connection(s)
I20260926 07:27:40.834586 14254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:40.835839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 14254
I20260926 07:27:40.835961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:27:40.836294 392 external_mini_cluster.cc:1428] Setting key 4889f3835448c4e8e524de30d748b0fd
Sep 26 07:27:40 dist-test-slave-g04g krb5kdc[14147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 07:27:40.849203 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:40.836639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33301 (local address 127.0.98.62:42159)
0926 07:27:40.836762 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:40.836780 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:40.837807 (+ 1027us) server_negotiation.cc:407] Connection header received
0926 07:27:40.839136 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:40.839139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:40.839192 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:40.839269 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:40.840403 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:40.840952 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:40.842074 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:40.842325 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:40.842439 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:40.845283 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:40.845288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:40.845291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:40.845314 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:40.846903 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:40.847492 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:40.847495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:40.847496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:40.847569 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:40.848034 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:40.848037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:40.848038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:40.848220 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:40.848412 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:40.848903 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 07:27:40.849027 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:27:40.850135 14385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:40.837138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:33301)
0926 07:27:40.837641 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:40.837703 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:40.838901 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:40.839474 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:40.839480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:40.839811 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:40.840255 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:40.840278 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:40.841077 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:40.841087 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:40.841953 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:40.841961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:40.842469 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:40.842472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:40.842643 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:40.843312 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:40.843328 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:40.845098 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:40.847019 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:40.847023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:40.847083 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:40.847401 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:40.847684 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:40.847686 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:40.847688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:40.847882 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:40.848410 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:40.848416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:40.848661 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 07:27:40.848848 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 07:27:40.849164 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":140,"threads_started":1}
I20260926 07:27:40.851675 14383 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42159
I20260926 07:27:40.851923 14383 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:40.852412 14383 heartbeater.cc:511] Master 127.0.98.62:42159 requested a full tablet report, sending...
I20260926 07:27:40.854118 14192 ts_manager.cc:194] Registered new tserver with Master: d0245ba57cdb486a9d045eff69a77388 (127.0.98.1:32917)
I20260926 07:27:40.855825 14192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:27:40.865785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42159
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:40.973915 14388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:40.974152 14388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:40.974192 14388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:40.978065 14388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:40.978113 14388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:40.978168 14388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:27:40.983263 14388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:40.984153 14388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:40.985069 14388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:40.994558 14396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:40.994777 14388 server_base.cc:1034] running on GCE node
W20260926 07:27:40.994594 14393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:40.994567 14394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:40.995456 14388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:40.996059 14388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:40.997320 14388 hybrid_clock.cc:648] HybridClock initialized: now 1790407660997302 us; error 31 us; skew 500 ppm
Sep 26 07:27:40 dist-test-slave-g04g krb5kdc[14147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:41.000008 14388 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:27:41.000864 14388 webserver.cc:460] Webserver started at http://127.0.98.2:33813/ using document root <none> and password file <none>
I20260926 07:27:41.001400 14388 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:41.001493 14388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:41.001719 14388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:41.003731 14388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "5c3b638bde800f1871c56444ca7f3686"
server_key_iv: "21466aaa8ba07b1d63c650e550061af8"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.004276 14388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "5c3b638bde800f1871c56444ca7f3686"
server_key_iv: "21466aaa8ba07b1d63c650e550061af8"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.007869 14388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:27:41.010244 14403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.011507 14388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:27:41.011637 14388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "5c3b638bde800f1871c56444ca7f3686"
server_key_iv: "21466aaa8ba07b1d63c650e550061af8"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.011744 14388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:41.030243 14388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:41.033233 14388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:41.033429 14388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:41.033994 14388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:41.034798 14388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:41.034868 14388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.034935 14388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:41.034988 14388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.044188 14388 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39873
I20260926 07:27:41.044206 14516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39873 every 8 connection(s)
I20260926 07:27:41.045032 14388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:41.051936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 14388
I20260926 07:27:41.052055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:27:41.052366 392 external_mini_cluster.cc:1428] Setting key 761149a1f4aa25325bef4e6ee0551cac
Sep 26 07:27:41 dist-test-slave-g04g krb5kdc[14147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:41.059016 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.046735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33141 (local address 127.0.98.62:42159)
0926 07:27:41.046885 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.046889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.047726 (+ 837us) server_negotiation.cc:407] Connection header received
0926 07:27:41.049014 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.049018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.049074 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.049180 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:41.050323 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.050856 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.052198 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.052517 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.052607 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.055498 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:41.055503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:41.055511 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:41.055534 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:41.056980 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.057487 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.057497 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.057499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.057583 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.057940 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.057943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.057945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.058124 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:41.058229 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.058719 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.058849 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:27:41.059769 14519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.047148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:33141)
0926 07:27:41.047576 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.047617 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:41.048794 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:41.049345 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:41.049352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:41.049713 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:41.050174 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.050187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.050982 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.050986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.052053 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.052065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.052672 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.052676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.052866 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.053500 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:41.053516 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:41.055336 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:41.057102 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:41.057107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:41.057121 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:41.057377 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:41.057686 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:41.057689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:41.057691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:41.057837 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:41.058239 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:41.058248 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:41.058485 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 07:27:41.058665 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 07:27:41.058971 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":111,"threads_started":1}
I20260926 07:27:41.061143 14517 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42159
I20260926 07:27:41.061403 14517 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 07:27:41.061929 14517 heartbeater.cc:511] Master 127.0.98.62:42159 requested a full tablet report, sending...
I20260926 07:27:41.062976 14192 ts_manager.cc:194] Registered new tserver with Master: 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873)
I20260926 07:27:41.063557 14192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:27:41.082108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42159
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:41.192405 14522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:41.192662 14522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:41.192740 14522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:41.196841 14522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:41.196914 14522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:41.196995 14522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:27:41.202173 14522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:41.203241 14522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:41.204217 14522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:41.213780 14528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:41.213778 14527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:41.213831 14530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:41.213904 14522 server_base.cc:1034] running on GCE node
I20260926 07:27:41.214414 14522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:41.215080 14522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:41.216238 14522 hybrid_clock.cc:648] HybridClock initialized: now 1790407661216224 us; error 27 us; skew 500 ppm
Sep 26 07:27:41 dist-test-slave-g04g krb5kdc[14147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:41.218950 14522 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:27:41.219760 14522 webserver.cc:460] Webserver started at http://127.0.98.3:35347/ using document root <none> and password file <none>
I20260926 07:27:41.220301 14522 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:41.220361 14522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:41.220567 14522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:41.222519 14522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "70847ca3aa2c46fcae60ad9c9332fea2"
format_stamp: "Formatted at 2026-09-26 07:27:41 on dist-test-slave-g04g"
server_key: "433f2d83d88fee334da7c7d8d67e79c5"
server_key_iv: "6f2823119d916a244c320433de8d7177"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.223124 14522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "70847ca3aa2c46fcae60ad9c9332fea2"
format_stamp: "Formatted at 2026-09-26 07:27:41 on dist-test-slave-g04g"
server_key: "433f2d83d88fee334da7c7d8d67e79c5"
server_key_iv: "6f2823119d916a244c320433de8d7177"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.226686 14522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:41.229019 14537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.230171 14522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:41.230306 14522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "70847ca3aa2c46fcae60ad9c9332fea2"
format_stamp: "Formatted at 2026-09-26 07:27:41 on dist-test-slave-g04g"
server_key: "433f2d83d88fee334da7c7d8d67e79c5"
server_key_iv: "6f2823119d916a244c320433de8d7177"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.230412 14522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:41.252399 14522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:41.255254 14522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:41.255470 14522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:41.256093 14522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:41.256932 14522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:41.257005 14522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.257071 14522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:41.257114 14522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.265300 14522 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45233
I20260926 07:27:41.265323 14650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45233 every 8 connection(s)
I20260926 07:27:41.266203 14522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:41.268427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 14522
I20260926 07:27:41.268540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:27:41.268821 392 external_mini_cluster.cc:1428] Setting key 691507a9f2a5c419678dedf2fc5453ef
Sep 26 07:27:41 dist-test-slave-g04g krb5kdc[14147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:41.280217 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.268078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55903 (local address 127.0.98.62:42159)
0926 07:27:41.268263 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.268269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.269151 (+ 882us) server_negotiation.cc:407] Connection header received
0926 07:27:41.270676 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.270679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.270723 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.270838 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:41.272123 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.272645 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.273727 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.273942 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.274020 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.276367 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:41.276374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:41.276376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:41.276400 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:41.277854 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.278492 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.278496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.278497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.278577 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.278998 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.279002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.279003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.279283 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:41.279372 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.279973 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.280094 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:27:41.281296 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.268462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:55903)
0926 07:27:41.268993 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.269049 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:41.270455 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:41.271079 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:41.271089 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:41.271516 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:41.271985 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.271998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.272748 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.272750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.273618 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.273626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.274066 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.274069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.274205 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.274734 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:41.274749 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:41.276206 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:41.277987 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:41.277993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:41.278006 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:41.278389 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:41.278706 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:41.278708 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:41.278710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:41.278862 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:41.279408 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:41.279414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:41.279701 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 07:27:41.279891 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 07:27:41.280194 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":149,"threads_started":1}
I20260926 07:27:41.282666 14651 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42159
I20260926 07:27:41.282946 14651 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:41.283485 14651 heartbeater.cc:511] Master 127.0.98.62:42159 requested a full tablet report, sending...
I20260926 07:27:41.284601 14192 ts_manager.cc:194] Registered new tserver with Master: 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233)
I20260926 07:27:41.285276 14192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55903
I20260926 07:27:41.294740 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:27:41.303445 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.296694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.0.98.62:42159)
0926 07:27:41.296818 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.296823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.296974 (+ 151us) server_negotiation.cc:407] Connection header received
0926 07:27:41.297074 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.297076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.297114 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.297187 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:41.297823 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.298379 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.299246 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.299452 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.299528 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.300743 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:41.300746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:41.300748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:41.300769 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:41.301960 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.302325 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.302328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.302329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.302397 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.302616 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.302618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.302619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.302809 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:41.302919 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.303136 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.303256 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:27:41.306207 14192 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:27:41.309204 14192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:41.323460 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.317397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39873 (local address 127.0.0.1:43878)
0926 07:27:41.318011 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.318066 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:41.318193 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:41.318796 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:41.318802 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:41.318826 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:41.318982 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.318989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.320214 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.320217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.321574 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.321584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.322823 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.322826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.322990 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.323052 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 07:27:41.323170 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 07:27:41.323291 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":27,"thread_start_us":107,"threads_started":1}
I20260926 07:27:41.323484 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.317582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.0.98.2:39873)
0926 07:27:41.318319 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.318330 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.318352 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:27:41.318428 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.318433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.318656 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.318818 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:41.319135 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.320062 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.321701 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.322482 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.322614 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.322762 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.323215 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.323327 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":269,"threads_started":1}
I20260926 07:27:41.324091 14662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.317654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32917 (local address 127.0.0.1:44024)
0926 07:27:41.318588 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.318602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:41.318687 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:41.319739 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:41.319742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:41.319759 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:41.319864 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.319873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.321282 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.321285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.322425 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.322434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.323695 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.323698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.323785 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.323807 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:27:41.323886 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:27:41.323967 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":93,"threads_started":1}
I20260926 07:27:41.324424 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.318719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.0.98.3:45233)
0926 07:27:41.319020 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.319053 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.320099 (+ 1046us) server_negotiation.cc:407] Connection header received
0926 07:27:41.320218 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.320223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.320376 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.320520 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:41.320697 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.321827 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.323099 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.323851 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.323995 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.324144 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.324199 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.324287 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":123,"threads_started":1}
I20260926 07:27:41.324497 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.318181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45233 (local address 127.0.0.1:56290)
0926 07:27:41.319995 (+ 1814us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.320006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:41.320113 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:41.320498 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:41.320501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:41.320518 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:41.320607 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.320612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.321970 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.321974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.322934 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.322942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.323989 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.323993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.324076 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.324097 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:27:41.324164 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 07:27:41.324361 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1751,"mutex_wait_us":23,"thread_start_us":87,"threads_started":1}
I20260926 07:27:41.325073 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.318366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.0.98.1:32917)
0926 07:27:41.319352 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.319368 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.319389 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:27:41.319462 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.319466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.319600 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.319755 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:41.319998 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.321142 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.322556 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.323359 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.323469 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.323633 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.324846 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.324943 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":108,"threads_started":1}
I20260926 07:27:41.326951 14585 tablet_service.cc:1467] Processing CreateTablet for tablet 75df1eff5cb7438dae92e9fd02be9166 (DEFAULT_TABLE table=test-table [id=663fc7e09ab34824a11bbad60fde9dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:41.326977 14451 tablet_service.cc:1467] Processing CreateTablet for tablet 75df1eff5cb7438dae92e9fd02be9166 (DEFAULT_TABLE table=test-table [id=663fc7e09ab34824a11bbad60fde9dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:41.327996 14585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75df1eff5cb7438dae92e9fd02be9166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:41.328019 14451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75df1eff5cb7438dae92e9fd02be9166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:41.327665 14317 tablet_service.cc:1467] Processing CreateTablet for tablet 75df1eff5cb7438dae92e9fd02be9166 (DEFAULT_TABLE table=test-table [id=663fc7e09ab34824a11bbad60fde9dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:41.328563 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75df1eff5cb7438dae92e9fd02be9166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:41.334159 14668 tablet_bootstrap.cc:492] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2: Bootstrap starting.
I20260926 07:27:41.336289 14668 tablet_bootstrap.cc:654] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:41.337365 14668 log.cc:824] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:41.337960 14669 tablet_bootstrap.cc:492] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Bootstrap starting.
I20260926 07:27:41.339366 14670 tablet_bootstrap.cc:492] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Bootstrap starting.
I20260926 07:27:41.339409 14668 tablet_bootstrap.cc:492] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2: No bootstrap required, opened a new log
I20260926 07:27:41.339584 14668 ts_tablet_manager.cc:1397] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:27:41.340513 14669 tablet_bootstrap.cc:654] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:41.341091 14670 tablet_bootstrap.cc:654] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:41.341305 14669 log.cc:824] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:41.342226 14670 log.cc:824] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:41.343354 14669 tablet_bootstrap.cc:492] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: No bootstrap required, opened a new log
I20260926 07:27:41.343537 14669 ts_tablet_manager.cc:1397] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260926 07:27:41.343459 14668 raft_consensus.cc:348] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } }
I20260926 07:27:41.343740 14668 raft_consensus.cc:374] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:41.343840 14668 raft_consensus.cc:729] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70847ca3aa2c46fcae60ad9c9332fea2, State: Initialized, Role: FOLLOWER
I20260926 07:27:41.344076 14670 tablet_bootstrap.cc:492] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: No bootstrap required, opened a new log
I20260926 07:27:41.344273 14670 ts_tablet_manager.cc:1397] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 07:27:41.344297 14668 consensus_queue.cc:260] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } }
I20260926 07:27:41.344950 14651 heartbeater.cc:503] Master 127.0.98.62:42159 was elected leader, sending a full tablet report...
I20260926 07:27:41.345271 14668 ts_tablet_manager.cc:1428] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:27:41.347275 14669 raft_consensus.cc:348] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } }
I20260926 07:27:41.347582 14669 raft_consensus.cc:374] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:41.347699 14669 raft_consensus.cc:729] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad0c20d79aa4f6a8e75f9b4b32938bd, State: Initialized, Role: FOLLOWER
I20260926 07:27:41.348134 14669 consensus_queue.cc:260] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } }
I20260926 07:27:41.348095 14670 raft_consensus.cc:348] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } }
I20260926 07:27:41.348395 14670 raft_consensus.cc:374] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:41.348511 14670 raft_consensus.cc:729] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0245ba57cdb486a9d045eff69a77388, State: Initialized, Role: FOLLOWER
I20260926 07:27:41.348755 14517 heartbeater.cc:503] Master 127.0.98.62:42159 was elected leader, sending a full tablet report...
I20260926 07:27:41.348958 14670 consensus_queue.cc:260] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } }
I20260926 07:27:41.349117 14669 ts_tablet_manager.cc:1428] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260926 07:27:41.349845 14383 heartbeater.cc:503] Master 127.0.98.62:42159 was elected leader, sending a full tablet report...
I20260926 07:27:41.349903 14670 ts_tablet_manager.cc:1428] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:27:41.479907 14674 raft_consensus.cc:482] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:41.480198 14674 raft_consensus.cc:504] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } }
I20260926 07:27:41.481405 14674 leader_election.cc:288] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873), d0245ba57cdb486a9d045eff69a77388 (127.0.98.1:32917)
I20260926 07:27:41.485368 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.481733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32917 (local address 127.0.98.3:38803)
0926 07:27:41.481953 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.481970 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:41.482086 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:41.482408 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:41.482411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:41.482429 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:41.482551 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.482557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.483377 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.483384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.484200 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.484210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.484897 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.484901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.485018 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.485038 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:27:41.485137 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:27:41.485227 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":1}
I20260926 07:27:41.485366 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.481835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38803 (local address 127.0.98.1:32917)
0926 07:27:41.481965 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.481970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.482170 (+ 200us) server_negotiation.cc:407] Connection header received
0926 07:27:41.482238 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.482240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.482288 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.482402 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:41.482677 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.483249 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.484347 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.484766 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.484898 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.484972 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.485152 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.485228 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:27:41.485507 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.481865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37137 (local address 127.0.98.2:39873)
0926 07:27:41.481996 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.482002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.482302 (+ 300us) server_negotiation.cc:407] Connection header received
0926 07:27:41.482464 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.482469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.482517 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.482605 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:41.482907 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.483459 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.484520 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.484944 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.485081 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.485142 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.485332 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.485401 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:27:41.485524 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.481733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39873 (local address 127.0.98.3:37137)
0926 07:27:41.482136 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.482147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:41.482350 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:41.482639 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:41.482642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:41.482657 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:41.482776 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.482781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.483597 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.483602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.484392 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.484401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.485089 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.485092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.485195 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.485216 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:27:41.485318 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:27:41.485383 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":63,"thread_start_us":111,"threads_started":1}
I20260926 07:27:41.486181 14337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75df1eff5cb7438dae92e9fd02be9166" candidate_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0245ba57cdb486a9d045eff69a77388" is_pre_election: true
I20260926 07:27:41.486279 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75df1eff5cb7438dae92e9fd02be9166" candidate_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" is_pre_election: true
I20260926 07:27:41.486522 14337 raft_consensus.cc:2454] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70847ca3aa2c46fcae60ad9c9332fea2 in term 0.
I20260926 07:27:41.486574 14471 raft_consensus.cc:2454] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70847ca3aa2c46fcae60ad9c9332fea2 in term 0.
I20260926 07:27:41.487018 14538 leader_election.cc:302] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad0c20d79aa4f6a8e75f9b4b32938bd, 70847ca3aa2c46fcae60ad9c9332fea2; no voters:
I20260926 07:27:41.487325 14674 raft_consensus.cc:2781] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:41.487411 14674 raft_consensus.cc:482] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:41.487453 14674 raft_consensus.cc:3037] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:41.488437 14674 raft_consensus.cc:504] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } }
I20260926 07:27:41.488833 14674 leader_election.cc:288] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [CANDIDATE]: Term 1 election: Requested vote from peers 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873), d0245ba57cdb486a9d045eff69a77388 (127.0.98.1:32917)
I20260926 07:27:41.489113 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75df1eff5cb7438dae92e9fd02be9166" candidate_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd"
I20260926 07:27:41.489259 14471 raft_consensus.cc:3037] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:41.489240 14337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75df1eff5cb7438dae92e9fd02be9166" candidate_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0245ba57cdb486a9d045eff69a77388"
I20260926 07:27:41.489395 14337 raft_consensus.cc:3037] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:41.490468 14471 raft_consensus.cc:2454] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70847ca3aa2c46fcae60ad9c9332fea2 in term 1.
I20260926 07:27:41.490623 14337 raft_consensus.cc:2454] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70847ca3aa2c46fcae60ad9c9332fea2 in term 1.
I20260926 07:27:41.490834 14538 leader_election.cc:302] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad0c20d79aa4f6a8e75f9b4b32938bd, 70847ca3aa2c46fcae60ad9c9332fea2; no voters:
I20260926 07:27:41.491067 14674 raft_consensus.cc:2781] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:41.491356 14674 raft_consensus.cc:686] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 1 LEADER]: Becoming Leader. State: Replica: 70847ca3aa2c46fcae60ad9c9332fea2, State: Running, Role: LEADER
I20260926 07:27:41.491707 14674 consensus_queue.cc:237] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } }
I20260926 07:27:41.495195 14192 catalog_manager.cc:5495] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } health_report { overall_health: HEALTHY } } }
I20260926 07:27:41.511613 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.507984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.0.98.3:45233)
0926 07:27:41.508101 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.508105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.508118 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:27:41.508254 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.508256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.508308 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.508425 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:41.508654 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.509157 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.510083 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.510315 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.510445 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.510725 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:41.511209 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:41.511334 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:41.511396 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 07:27:41.511448 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 07:27:41.517160 14471 raft_consensus.cc:1261] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 1 FOLLOWER]: Refusing update from remote peer 70847ca3aa2c46fcae60ad9c9332fea2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 07:27:41.517174 14652 tablet.cc:2369] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:41.517198 14337 raft_consensus.cc:1261] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Refusing update from remote peer 70847ca3aa2c46fcae60ad9c9332fea2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:41.517872 14678 consensus_queue.cc:1035] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:41.518126 14674 consensus_queue.cc:1035] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:41.524211 14681 mvcc.cc:205] Tried to move back new op lower bound from 7333509781567758336 to 7333509781473234944. Current Snapshot: MvccSnapshot[applied={T|T < 7333509781567758336}]
I20260926 07:27:41.527500 14682 mvcc.cc:205] Tried to move back new op lower bound from 7333509781567758336 to 7333509781473234944. Current Snapshot: MvccSnapshot[applied={T|T < 7333509781567758336}]
I20260926 07:27:41.534003 14191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:41.534168 14191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:41.536412 14191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Sending DeleteTablet for 3 replicas of tablet 75df1eff5cb7438dae92e9fd02be9166
I20260926 07:27:41.537125 14317 tablet_service.cc:1514] Processing DeleteTablet for tablet 75df1eff5cb7438dae92e9fd02be9166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44024
I20260926 07:27:41.537184 14451 tablet_service.cc:1514] Processing DeleteTablet for tablet 75df1eff5cb7438dae92e9fd02be9166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43878
I20260926 07:27:41.537585 14691 tablet_replica.cc:330] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: stopping tablet replica
I20260926 07:27:41.537621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 14254
I20260926 07:27:41.537611 14692 tablet_replica.cc:330] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: stopping tablet replica
I20260926 07:27:41.537971 14691 raft_consensus.cc:2229] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:41.538025 14692 raft_consensus.cc:2229] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:41.538448 14692 raft_consensus.cc:2258] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:41.539878 14692 ts_tablet_manager.cc:1905] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:41.541189 14585 tablet_service.cc:1514] Processing DeleteTablet for tablet 75df1eff5cb7438dae92e9fd02be9166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56290
I20260926 07:27:41.542570 14692 ts_tablet_manager.cc:1918] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:41.542667 14692 log.cc:1196] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/75df1eff5cb7438dae92e9fd02be9166
I20260926 07:27:41.542994 14692 ts_tablet_manager.cc:1939] T 75df1eff5cb7438dae92e9fd02be9166 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Deleting consensus metadata
I20260926 07:27:41.544234 14693 tablet_replica.cc:330] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2: stopping tablet replica
I20260926 07:27:41.544256 14176 catalog_manager.cc:4841] TS 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873): tablet 75df1eff5cb7438dae92e9fd02be9166 (table test-table [id=663fc7e09ab34824a11bbad60fde9dfc]) successfully deleted
I20260926 07:27:41.544518 14693 raft_consensus.cc:2229] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:27:41.544896 14693 raft_consensus.cc:2258] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:41.546067 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 14388
W20260926 07:27:41.546062 14177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32917 (error 108)
I20260926 07:27:41.546132 14693 ts_tablet_manager.cc:1905] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:41.548787 14693 ts_tablet_manager.cc:1918] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:41.548866 14693 log.cc:1196] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/75df1eff5cb7438dae92e9fd02be9166
I20260926 07:27:41.549167 14693 ts_tablet_manager.cc:1939] T 75df1eff5cb7438dae92e9fd02be9166 P 70847ca3aa2c46fcae60ad9c9332fea2: Deleting consensus metadata
I20260926 07:27:41.550230 14176 catalog_manager.cc:4841] TS 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233): tablet 75df1eff5cb7438dae92e9fd02be9166 (table test-table [id=663fc7e09ab34824a11bbad60fde9dfc]) successfully deleted
I20260926 07:27:41.551918 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.551609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32917 (local address 127.0.0.1:44036)
0926 07:27:41.551760 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.551833 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 07:27:41.552115 14177 catalog_manager.cc:4584] TS d0245ba57cdb486a9d045eff69a77388 (127.0.98.1:32917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75df1eff5cb7438dae92e9fd02be9166: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32917: connect: Connection refused (error 111)
I20260926 07:27:41.552217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 14522
I20260926 07:27:41.557824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 14160
I20260926 07:27:41.564270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42159
--webserver_interface=127.0.98.62
--webserver_port=37935
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:41.677289 14695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:41.677553 14695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:41.677593 14695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:41.681635 14695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:41.681694 14695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:41.681713 14695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:41.681730 14695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:41.681746 14695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:41.686921 14695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:41.688031 14695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:41.689061 14695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:41.702644 14702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:41.702638 14701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:41.702827 14695 server_base.cc:1034] running on GCE node
W20260926 07:27:41.702646 14704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:41.703536 14695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:41.704649 14695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:41.705837 14695 hybrid_clock.cc:648] HybridClock initialized: now 1790407661705801 us; error 52 us; skew 500 ppm
Sep 26 07:27:41 dist-test-slave-g04g krb5kdc[14147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:41.708593 14695 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:27:41.709486 14695 webserver.cc:460] Webserver started at http://127.0.98.62:37935/ using document root <none> and password file <none>
I20260926 07:27:41.710006 14695 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:41.710071 14695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:41.713861 14695 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:41.715963 14711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.717167 14695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:41.717264 14695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "eebffc3380404166a99eb1ae392d0814"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "5fcae58da9f843d3cb844e709e86df49"
server_key_iv: "cf565fce1eea765bb9b83c0726e2498d"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.717795 14695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:41.757581 14695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:41.760600 14695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:41.760757 14695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:41.767741 14695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42159
I20260926 07:27:41.768054 14763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42159 every 8 connection(s)
I20260926 07:27:41.768855 14695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:41.770864 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 14695
I20260926 07:27:41.771387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32917
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42159
--webserver_port=33165
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:27:41.773885 14764 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:27:41.775686 14764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Bootstrap starting.
I20260926 07:27:41.792395 14764 log.cc:824] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:41.801164 14764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:27:41.801551 14764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Bootstrap complete.
I20260926 07:27:41.805588 14764 raft_consensus.cc:348] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } }
I20260926 07:27:41.805972 14764 raft_consensus.cc:729] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebffc3380404166a99eb1ae392d0814, State: Initialized, Role: FOLLOWER
I20260926 07:27:41.806449 14764 consensus_queue.cc:260] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } }
I20260926 07:27:41.806558 14764 raft_consensus.cc:388] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:41.806651 14764 raft_consensus.cc:482] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:41.806766 14764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:27:41.808660 14764 raft_consensus.cc:504] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } }
I20260926 07:27:41.808990 14764 leader_election.cc:302] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eebffc3380404166a99eb1ae392d0814; no voters:
I20260926 07:27:41.809315 14764 leader_election.cc:288] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:27:41.809427 14768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:27:41.809667 14768 raft_consensus.cc:686] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 2 LEADER]: Becoming Leader. State: Replica: eebffc3380404166a99eb1ae392d0814, State: Running, Role: LEADER
I20260926 07:27:41.809948 14768 consensus_queue.cc:237] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } }
I20260926 07:27:41.810482 14764 sys_catalog.cc:564] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:41.810886 14770 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eebffc3380404166a99eb1ae392d0814. Latest consensus state: current_term: 2 leader_uuid: "eebffc3380404166a99eb1ae392d0814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } } }
I20260926 07:27:41.810999 14770 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:41.811378 14772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:41.811386 14769 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eebffc3380404166a99eb1ae392d0814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } } }
I20260926 07:27:41.811519 14769 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:41.816275 14772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:41.816876 14772 catalog_manager.cc:1259] Loaded cluster ID: d65d00b38d01403f8b9fccb639daeb4f
I20260926 07:27:41.816933 14772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:41.818256 14772 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 07:27:41.822885 14772 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 07:27:41.828231 14772 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:41.828972 14772 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Loaded TSK: 0
I20260926 07:27:41.830123 14772 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:27:41.887730 14766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:41.887986 14766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:41.888031 14766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:41.892057 14766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:41.892113 14766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:41.892170 14766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:27:41.897837 14766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33165
--tserver_master_addrs=127.0.98.62:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:41.898828 14766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:41.899871 14766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:41.909555 14791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:41.909582 14794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:41.909597 14792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:41.909889 14766 server_base.cc:1034] running on GCE node
I20260926 07:27:41.910264 14766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:41.910981 14766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:41.912159 14766 hybrid_clock.cc:648] HybridClock initialized: now 1790407661912133 us; error 43 us; skew 500 ppm
Sep 26 07:27:41 dist-test-slave-g04g krb5kdc[14147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:41.914861 14766 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:27:41.915791 14766 webserver.cc:460] Webserver started at http://127.0.98.1:33165/ using document root <none> and password file <none>
I20260926 07:27:41.916353 14766 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:41.916425 14766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:41.920456 14766 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 07:27:41.922704 14801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:41.924042 14766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:41.924185 14766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "d0245ba57cdb486a9d045eff69a77388"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "62a3d9a97e62eec2cf0ef41afd629ad7"
server_key_iv: "2ef8c00b34bfac0ea7354dc63c82db55"
server_key_version: "encryptionkey@0"
I20260926 07:27:41.924723 14766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:41.959451 14766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:41.962602 14766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:41.962812 14766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:41.963439 14766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:41.964516 14808 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:27:41.968734 14766 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:27:41.968806 14766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 07:27:41.968876 14766 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:27:41.970436 14766 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:27:41.970505 14766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 07:27:41.970726 14808 tablet_bootstrap.cc:492] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Bootstrap starting.
I20260926 07:27:41.980799 14766 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32917
I20260926 07:27:41.980909 14915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32917 every 8 connection(s)
I20260926 07:27:41.981772 14766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:41.988283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 14766
I20260926 07:27:41.988754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39873
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42159
--webserver_port=33813
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:27:41.990329 14808 log.cc:824] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:27:41 dist-test-slave-g04g krb5kdc[14147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:42.000083 14808 tablet_bootstrap.cc:492] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:27:42.000582 14808 tablet_bootstrap.cc:492] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Bootstrap complete.
I20260926 07:27:42.001222 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.983865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33579 (local address 127.0.98.62:42159)
0926 07:27:41.984298 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:41.984317 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:41.984422 (+ 105us) server_negotiation.cc:407] Connection header received
0926 07:27:41.985996 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:41.986009 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:41.986307 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:41.986572 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:41.988324 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.989868 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.991538 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.992002 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.992123 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.996251 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:41.996257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:41.996269 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:41.996297 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:41.998118 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.998638 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.998643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.998647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.998756 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:41.999131 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:41.999134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:41.999136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:41.999455 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:41.999592 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.000125 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.000483 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":103,"threads_started":1}
I20260926 07:27:42.001233 14918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:41.983712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:33579)
0926 07:27:41.984260 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:41.984318 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:41.985572 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:41.986753 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:41.986763 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:41.987285 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:41.988104 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.988127 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.990079 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.990084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.991376 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:41.991388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:41.992150 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:41.992155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:41.992349 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:41.994336 (+ 1987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:41.994352 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:41.996060 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:41.998234 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:41.998239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:41.998251 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:41.998510 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:41.998860 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:41.998865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:41.998867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:41.999003 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:41.999577 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:41.999583 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:41.999828 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 07:27:42.000004 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 07:27:42.000345 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":145,"threads_started":1}
I20260926 07:27:42.002102 14808 ts_tablet_manager.cc:1397] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.000s
I20260926 07:27:42.003710 14916 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42159
I20260926 07:27:42.003931 14916 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:42.004451 14916 heartbeater.cc:511] Master 127.0.98.62:42159 requested a full tablet report, sending...
I20260926 07:27:42.005236 14808 raft_consensus.cc:348] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } }
I20260926 07:27:42.005754 14808 raft_consensus.cc:729] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0245ba57cdb486a9d045eff69a77388, State: Initialized, Role: FOLLOWER
I20260926 07:27:42.006264 14808 consensus_queue.cc:260] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } }
I20260926 07:27:42.007133 14728 ts_manager.cc:194] Registered new tserver with Master: d0245ba57cdb486a9d045eff69a77388 (127.0.98.1:32917)
I20260926 07:27:42.007301 14808 ts_tablet_manager.cc:1428] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260926 07:27:42.010571 14728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33579
I20260926 07:27:42.012897 14916 heartbeater.cc:503] Master 127.0.98.62:42159 was elected leader, sending a full tablet report...
Sep 26 07:27:42 dist-test-slave-g04g krb5kdc[14147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 07:27:42.022065 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.009772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32917 (local address 127.0.0.1:44040)
0926 07:27:42.010155 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:42.010198 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:42.010320 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:42.011011 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:42.011017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:42.011500 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:42.013253 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.013267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.014562 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.014564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.015241 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.015251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.015991 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.016003 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.016078 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.016621 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:42.016632 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:42.017951 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:42.020283 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:42.020292 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:42.020296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:42.020522 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:42.020854 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:42.020856 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:42.020858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:42.020932 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:42.021493 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:42.021497 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:42.021629 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 07:27:42.021781 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 07:27:42.021941 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":102,"threads_started":1}
I20260926 07:27:42.022231 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.009828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.0.98.1:32917)
0926 07:27:42.010529 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.010542 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.010566 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:27:42.010631 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.010636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.010819 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.010963 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:42.013358 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.014412 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.015358 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.015859 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.015996 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.018086 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:42.018102 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:42.018114 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:42.018148 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:42.020081 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.020649 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.020655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.020658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.020759 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.021023 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.021027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.021029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.021339 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:42.021532 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.021865 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.022035 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":103,"threads_started":1}
I20260926 07:27:42.023195 14850 tablet_service.cc:1514] Processing DeleteTablet for tablet 75df1eff5cb7438dae92e9fd02be9166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44040
I20260926 07:27:42.023973 14925 tablet_replica.cc:330] stopping tablet replica
I20260926 07:27:42.024245 14925 raft_consensus.cc:2229] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:42.024483 14925 raft_consensus.cc:2258] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:42.025389 14925 ts_tablet_manager.cc:1905] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:42.028820 14925 ts_tablet_manager.cc:1918] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:42.028925 14925 log.cc:1196] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal/wals/75df1eff5cb7438dae92e9fd02be9166
I20260926 07:27:42.029209 14925 ts_tablet_manager.cc:1939] T 75df1eff5cb7438dae92e9fd02be9166 P d0245ba57cdb486a9d045eff69a77388: Deleting consensus metadata
I20260926 07:27:42.030784 14713 catalog_manager.cc:4841] TS d0245ba57cdb486a9d045eff69a77388 (127.0.98.1:32917): tablet 75df1eff5cb7438dae92e9fd02be9166 (table test-table [id=663fc7e09ab34824a11bbad60fde9dfc]) successfully deleted
W20260926 07:27:42.110419 14920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:42.110687 14920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:42.110735 14920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:42.114853 14920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:42.114917 14920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:42.114976 14920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:27:42.120265 14920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33813
--tserver_master_addrs=127.0.98.62:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:42.121328 14920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:42.122357 14920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:42.132040 14931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:42.132035 14930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:42.132063 14933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:42.132236 14920 server_base.cc:1034] running on GCE node
I20260926 07:27:42.132700 14920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:42.133319 14920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:42.134487 14920 hybrid_clock.cc:648] HybridClock initialized: now 1790407662134460 us; error 39 us; skew 500 ppm
Sep 26 07:27:42 dist-test-slave-g04g krb5kdc[14147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:42.137157 14920 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:27:42.138054 14920 webserver.cc:460] Webserver started at http://127.0.98.2:33813/ using document root <none> and password file <none>
I20260926 07:27:42.138608 14920 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:42.138703 14920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:42.142422 14920 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:42.144385 14940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:42.145524 14920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 07:27:42.145656 14920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "5c3b638bde800f1871c56444ca7f3686"
server_key_iv: "21466aaa8ba07b1d63c650e550061af8"
server_key_version: "encryptionkey@0"
I20260926 07:27:42.146166 14920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:42.169739 14920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:42.172624 14920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:42.172833 14920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:42.173430 14920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:42.174256 14920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:42.174329 14920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:42.174391 14920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:42.174440 14920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:42.183269 14920 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39873
I20260926 07:27:42.183301 15053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39873 every 8 connection(s)
I20260926 07:27:42.184250 14920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:42.187206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 14920
I20260926 07:27:42.187707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45233
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42159
--webserver_port=35347
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:27:42 dist-test-slave-g04g krb5kdc[14147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:42.199584 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.185951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37641 (local address 127.0.98.62:42159)
0926 07:27:42.186112 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.186117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.186938 (+ 821us) server_negotiation.cc:407] Connection header received
0926 07:27:42.188826 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.188829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.188881 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.188956 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:42.190228 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.191156 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.192377 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.192827 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.192964 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.195778 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:42.195781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:42.195783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:42.195808 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:42.197332 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.197978 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.197981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.197982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.198050 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.198457 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.198459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.198460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.198615 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:42.198688 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.199349 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.199453 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:27:42.200610 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.186353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:37641)
0926 07:27:42.186769 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:42.186810 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:42.188588 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:42.189198 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:42.189210 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:42.189588 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:42.190058 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.190071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.191280 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.191283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.192247 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.192256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.192978 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.192980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.193113 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.193724 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:42.193739 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:42.195604 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:42.197454 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:42.197462 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:42.197503 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:42.197881 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:42.198142 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:42.198148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:42.198152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:42.198357 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:42.198723 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:42.198731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:42.199099 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 07:27:42.199316 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 07:27:42.199630 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":105,"threads_started":1}
I20260926 07:27:42.201735 15054 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42159
I20260926 07:27:42.201975 15054 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:42.202466 15054 heartbeater.cc:511] Master 127.0.98.62:42159 requested a full tablet report, sending...
I20260926 07:27:42.203469 14728 ts_manager.cc:194] Registered new tserver with Master: 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873)
I20260926 07:27:42.204021 14728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37641
W20260926 07:27:42.299275 15057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:42.299546 15057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:42.299618 15057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:42.303606 15057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:42.303679 15057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:42.303769 15057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:27:42.308835 15057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35347
--tserver_master_addrs=127.0.98.62:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:42.309851 15057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:42.310930 15057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:42.320500 15063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:42.320515 15062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:42.320482 15065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:42.320747 15057 server_base.cc:1034] running on GCE node
I20260926 07:27:42.321179 15057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:42.321784 15057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:42.322985 15057 hybrid_clock.cc:648] HybridClock initialized: now 1790407662322972 us; error 24 us; skew 500 ppm
Sep 26 07:27:42 dist-test-slave-g04g krb5kdc[14147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:42.325527 15057 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:27:42.326341 15057 webserver.cc:460] Webserver started at http://127.0.98.3:35347/ using document root <none> and password file <none>
I20260926 07:27:42.326841 15057 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:42.326910 15057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:42.330875 15057 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:27:42.332902 15072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:42.334251 15057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:42.334354 15057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "70847ca3aa2c46fcae60ad9c9332fea2"
format_stamp: "Formatted at 2026-09-26 07:27:41 on dist-test-slave-g04g"
server_key: "433f2d83d88fee334da7c7d8d67e79c5"
server_key_iv: "6f2823119d916a244c320433de8d7177"
server_key_version: "encryptionkey@0"
I20260926 07:27:42.334846 15057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:42.343370 15057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:42.346361 15057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:42.346494 15057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:42.346959 15057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:42.347724 15057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:42.347777 15057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:42.347813 15057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:42.347828 15057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:42.356480 15057 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45233
I20260926 07:27:42.356504 15185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45233 every 8 connection(s)
I20260926 07:27:42.357438 15057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:42.364095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 15057
Sep 26 07:27:42 dist-test-slave-g04g krb5kdc[14147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:42.371717 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.359223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46211 (local address 127.0.98.62:42159)
0926 07:27:42.359367 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.359372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.360232 (+ 860us) server_negotiation.cc:407] Connection header received
0926 07:27:42.361467 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.361470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.361516 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.361595 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:42.362715 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.363387 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.364583 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.364873 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.364994 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.367789 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:42.367793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:42.367796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:42.367829 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:42.369336 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.369848 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.369852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.369854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.369945 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.370392 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.370398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.370400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.370694 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:42.370829 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.371417 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.371540 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:27:42.372633 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.359679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:46211)
0926 07:27:42.360090 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:42.360129 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:42.361242 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:42.361752 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:42.361760 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:42.362096 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:42.362557 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.362569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.363497 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.363500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.364449 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.364458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.365114 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.365119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.365327 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.365940 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:42.365964 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:42.367621 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:42.369453 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:42.369459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:42.369472 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:42.369738 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:42.370036 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:42.370040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:42.370042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:42.370302 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:42.370826 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:42.370833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:42.371138 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 07:27:42.371360 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 07:27:42.371691 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":110,"threads_started":1}
I20260926 07:27:42.372947 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.364814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.0.98.62:42159)
0926 07:27:42.365044 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.365049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.365202 (+ 153us) server_negotiation.cc:407] Connection header received
0926 07:27:42.365414 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.365417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.365460 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.365539 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:42.366496 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.366996 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.367962 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.368308 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.368422 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.369854 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:42.369858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:42.369860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:42.369882 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:42.371376 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.371808 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.371812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.371816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.371890 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:42.372156 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:42.372161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:42.372163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:42.372331 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:42.372536 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.372673 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.372781 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260926 07:27:42.373812 15186 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42159
I20260926 07:27:42.374081 15186 heartbeater.cc:464] Registering TS with master...
I20260926 07:27:42.374600 15186 heartbeater.cc:511] Master 127.0.98.62:42159 requested a full tablet report, sending...
I20260926 07:27:42.375685 14727 ts_manager.cc:194] Registered new tserver with Master: 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233)
I20260926 07:27:42.376245 14727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46211
I20260926 07:27:42.386556 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:27:42.390606 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.387134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.0.98.62:42159)
0926 07:27:42.387247 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.387251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.387487 (+ 236us) server_negotiation.cc:407] Connection header received
0926 07:27:42.387620 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.387622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.387714 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.387797 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:42.388001 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.388494 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.389397 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.389609 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.389684 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.389946 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:42.390321 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:42.390405 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.390465 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.390512 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:27:42.391275 14727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:27:42.394570 14727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:27:42.402442 14850 tablet_service.cc:1467] Processing CreateTablet for tablet 625049587e924a2ca9f7cff47e3f6165 (DEFAULT_TABLE table=test-table [id=77f00fceafc24b309273830ca3e122dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:42.403126 14850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625049587e924a2ca9f7cff47e3f6165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:42.405869 14808 tablet_bootstrap.cc:492] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388: Bootstrap starting.
I20260926 07:27:42.406777 14808 tablet_bootstrap.cc:654] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:42.407097 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.401339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39873 (local address 127.0.0.1:43886)
0926 07:27:42.401454 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:42.401469 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:42.401540 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:42.402599 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:42.402602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:42.402618 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:42.402718 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.402725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.404192 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.404195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.405452 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.405464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.406622 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.406625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.406719 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.406761 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 07:27:42.406838 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:27:42.406924 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":33}
I20260926 07:27:42.407413 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.401922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.0.98.3:45233)
0926 07:27:42.402390 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.402401 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.402425 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:27:42.402502 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.402508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.402844 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.403012 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:42.403259 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.404315 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.405543 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.406332 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.406437 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.406580 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.407124 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.407259 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":157,"threads_started":1}
I20260926 07:27:42.407557 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.401733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.0.98.2:39873)
0926 07:27:42.402081 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.402094 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.402120 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:27:42.402215 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.402224 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.402453 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.402625 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:42.402823 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.404037 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.405604 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.406473 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.406622 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.406758 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.407265 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.407374 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":155,"threads_started":1}
I20260926 07:27:42.407811 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.401589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45233 (local address 127.0.0.1:56304)
0926 07:27:42.402180 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:42.402193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:42.402272 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:42.403007 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:42.403009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:42.403048 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:42.403130 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.403136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.404469 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.404472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.405418 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.405427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.406646 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.406653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.406771 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.406836 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 07:27:42.407576 (+ 740us) client_negotiation.cc:240] Negotiation successful
0926 07:27:42.407672 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":51,"thread_start_us":132,"threads_started":1}
I20260926 07:27:42.408757 14808 tablet_bootstrap.cc:492] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388: No bootstrap required, opened a new log
I20260926 07:27:42.408875 14808 ts_tablet_manager.cc:1397] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:27:42.409278 14808 raft_consensus.cc:348] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:42.409401 14808 raft_consensus.cc:374] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:42.409446 14808 raft_consensus.cc:729] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0245ba57cdb486a9d045eff69a77388, State: Initialized, Role: FOLLOWER
I20260926 07:27:42.409600 14808 consensus_queue.cc:260] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:42.409926 14808 ts_tablet_manager.cc:1428] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:27:42.410696 15120 tablet_service.cc:1467] Processing CreateTablet for tablet 625049587e924a2ca9f7cff47e3f6165 (DEFAULT_TABLE table=test-table [id=77f00fceafc24b309273830ca3e122dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:42.410701 14988 tablet_service.cc:1467] Processing CreateTablet for tablet 625049587e924a2ca9f7cff47e3f6165 (DEFAULT_TABLE table=test-table [id=77f00fceafc24b309273830ca3e122dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:27:42.411834 15120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625049587e924a2ca9f7cff47e3f6165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:42.411837 14988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625049587e924a2ca9f7cff47e3f6165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:42.420413 15200 tablet_bootstrap.cc:492] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Bootstrap starting.
I20260926 07:27:42.420286 15199 tablet_bootstrap.cc:492] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2: Bootstrap starting.
I20260926 07:27:42.423167 15199 tablet_bootstrap.cc:654] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:42.423657 15200 tablet_bootstrap.cc:654] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:42.424495 15200 log.cc:824] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:42.424495 15199 log.cc:824] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:42.426476 15199 tablet_bootstrap.cc:492] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2: No bootstrap required, opened a new log
I20260926 07:27:42.426473 15200 tablet_bootstrap.cc:492] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: No bootstrap required, opened a new log
I20260926 07:27:42.426671 15200 ts_tablet_manager.cc:1397] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260926 07:27:42.426682 15199 ts_tablet_manager.cc:1397] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260926 07:27:42.429833 15200 raft_consensus.cc:348] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:42.429833 15199 raft_consensus.cc:348] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:42.430066 15200 raft_consensus.cc:374] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:42.430092 15199 raft_consensus.cc:374] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:42.430130 15200 raft_consensus.cc:729] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad0c20d79aa4f6a8e75f9b4b32938bd, State: Initialized, Role: FOLLOWER
I20260926 07:27:42.430186 15199 raft_consensus.cc:729] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70847ca3aa2c46fcae60ad9c9332fea2, State: Initialized, Role: FOLLOWER
I20260926 07:27:42.430648 15200 consensus_queue.cc:260] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:42.430648 15199 consensus_queue.cc:260] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:42.431303 15054 heartbeater.cc:503] Master 127.0.98.62:42159 was elected leader, sending a full tablet report...
I20260926 07:27:42.431303 15186 heartbeater.cc:503] Master 127.0.98.62:42159 was elected leader, sending a full tablet report...
I20260926 07:27:42.431656 15200 ts_tablet_manager.cc:1428] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 07:27:42.431656 15199 ts_tablet_manager.cc:1428] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260926 07:27:42.435123 15055 tablet.cc:2369] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:42.463282 15203 raft_consensus.cc:482] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:42.463485 15203 raft_consensus.cc:504] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:42.464632 15203 leader_election.cc:288] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873), d0245ba57cdb486a9d045eff69a77388 (127.0.98.1:32917)
I20260926 07:27:42.468914 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.464870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39873 (local address 127.0.98.3:33067)
0926 07:27:42.465071 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:42.465086 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:42.465172 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:42.465551 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:42.465556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:42.465582 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:42.465709 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.465716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.466714 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.466717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.467715 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.467724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.468483 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.468486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.468592 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.468618 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:27:42.468700 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 07:27:42.468772 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 07:27:42.468927 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.465089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33067 (local address 127.0.98.2:39873)
0926 07:27:42.465296 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.465305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.465317 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:27:42.465362 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.465364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.465403 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.465507 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:42.465843 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.466605 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.467840 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.468369 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.468448 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.468496 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.468720 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.468800 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 07:27:42.469331 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.465078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42057 (local address 127.0.98.1:32917)
0926 07:27:42.465207 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.465214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.465909 (+ 695us) server_negotiation.cc:407] Connection header received
0926 07:27:42.465992 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.465994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.466050 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.466128 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:27:42.466398 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.467082 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.468169 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.468733 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.468826 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.468985 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.469134 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.469204 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:27:42.469705 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.464870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32917 (local address 127.0.98.3:42057)
0926 07:27:42.465807 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:42.465821 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:42.465915 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:42.466163 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:42.466165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:42.466182 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:27:42.466290 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.466296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.467188 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.467191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.468047 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:42.468057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.468848 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.468852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:42.468969 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.468999 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:27:42.469489 (+ 490us) client_negotiation.cc:240] Negotiation successful
0926 07:27:42.469563 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":558,"thread_start_us":147,"threads_started":1}
I20260926 07:27:42.469663 15008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625049587e924a2ca9f7cff47e3f6165" candidate_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" is_pre_election: true
I20260926 07:27:42.470033 15008 raft_consensus.cc:2454] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70847ca3aa2c46fcae60ad9c9332fea2 in term 0.
I20260926 07:27:42.470364 14870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625049587e924a2ca9f7cff47e3f6165" candidate_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0245ba57cdb486a9d045eff69a77388" is_pre_election: true
I20260926 07:27:42.470537 15073 leader_election.cc:302] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad0c20d79aa4f6a8e75f9b4b32938bd, 70847ca3aa2c46fcae60ad9c9332fea2; no voters:
I20260926 07:27:42.470666 14870 raft_consensus.cc:2454] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70847ca3aa2c46fcae60ad9c9332fea2 in term 0.
I20260926 07:27:42.470855 15203 raft_consensus.cc:2781] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:27:42.470973 15203 raft_consensus.cc:482] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:27:42.471058 15203 raft_consensus.cc:3037] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:42.472090 15203 raft_consensus.cc:504] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:42.472489 15203 leader_election.cc:288] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [CANDIDATE]: Term 1 election: Requested vote from peers 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873), d0245ba57cdb486a9d045eff69a77388 (127.0.98.1:32917)
I20260926 07:27:42.472831 15008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625049587e924a2ca9f7cff47e3f6165" candidate_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd"
I20260926 07:27:42.472831 14870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625049587e924a2ca9f7cff47e3f6165" candidate_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0245ba57cdb486a9d045eff69a77388"
I20260926 07:27:42.472951 15008 raft_consensus.cc:3037] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:42.472951 14870 raft_consensus.cc:3037] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:42.474200 14870 raft_consensus.cc:2454] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70847ca3aa2c46fcae60ad9c9332fea2 in term 1.
I20260926 07:27:42.474200 15008 raft_consensus.cc:2454] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70847ca3aa2c46fcae60ad9c9332fea2 in term 1.
I20260926 07:27:42.474462 15073 leader_election.cc:302] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad0c20d79aa4f6a8e75f9b4b32938bd, 70847ca3aa2c46fcae60ad9c9332fea2; no voters:
I20260926 07:27:42.474612 15203 raft_consensus.cc:2781] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:42.474839 15203 raft_consensus.cc:686] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 1 LEADER]: Becoming Leader. State: Replica: 70847ca3aa2c46fcae60ad9c9332fea2, State: Running, Role: LEADER
I20260926 07:27:42.475221 15203 consensus_queue.cc:237] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:42.478428 14727 catalog_manager.cc:5495] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } health_report { overall_health: UNKNOWN } } }
W20260926 07:27:42.482939 14917 tablet.cc:2369] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:27:42.488065 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.484931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.0.98.3:45233)
0926 07:27:42.485051 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:42.485054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:42.485069 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:27:42.485210 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:42.485214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:42.485266 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:42.485351 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:27:42.485532 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.486021 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.486817 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:42.487023 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:42.487178 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:42.487414 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:27:42.487791 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:27:42.487876 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:42.487930 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 07:27:42.487973 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:27:42.493338 14870 raft_consensus.cc:1261] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Refusing update from remote peer 70847ca3aa2c46fcae60ad9c9332fea2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:42.493358 15008 raft_consensus.cc:1261] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 1 FOLLOWER]: Refusing update from remote peer 70847ca3aa2c46fcae60ad9c9332fea2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:27:42.494053 15206 consensus_queue.cc:1035] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:42.494288 15203 consensus_queue.cc:1035] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:27:42.500849 15210 mvcc.cc:205] Tried to move back new op lower bound from 7333509785566699520 to 7333509785501163520. Current Snapshot: MvccSnapshot[applied={T|T < 7333509785566699520}]
I20260926 07:27:42.503114 15209 mvcc.cc:205] Tried to move back new op lower bound from 7333509785566699520 to 7333509785501163520. Current Snapshot: MvccSnapshot[applied={T|T < 7333509785566699520}]
I20260926 07:27:42.508980 14727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:42.509135 14727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:27:42.511535 14727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Sending DeleteTablet for 3 replicas of tablet 625049587e924a2ca9f7cff47e3f6165
I20260926 07:27:42.512351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 14766
I20260926 07:27:42.512421 14988 tablet_service.cc:1514] Processing DeleteTablet for tablet 625049587e924a2ca9f7cff47e3f6165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43886
I20260926 07:27:42.512427 15120 tablet_service.cc:1514] Processing DeleteTablet for tablet 625049587e924a2ca9f7cff47e3f6165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:27:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56304
I20260926 07:27:42.513860 15219 tablet_replica.cc:330] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: stopping tablet replica
I20260926 07:27:42.514178 15219 raft_consensus.cc:2229] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:27:42.514470 15219 raft_consensus.cc:2258] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:27:42.516050 15219 ts_tablet_manager.cc:1905] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:42.519212 15218 tablet_replica.cc:330] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2: stopping tablet replica
I20260926 07:27:42.519459 15219 ts_tablet_manager.cc:1918] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:42.519575 15219 log.cc:1196] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/625049587e924a2ca9f7cff47e3f6165
I20260926 07:27:42.519604 15218 raft_consensus.cc:2229] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:27:42.519928 15219 ts_tablet_manager.cc:1939] T 625049587e924a2ca9f7cff47e3f6165 P 2ad0c20d79aa4f6a8e75f9b4b32938bd: Deleting consensus metadata
I20260926 07:27:42.520151 15218 raft_consensus.cc:2258] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 07:27:42.520629 14713 connection.cc:535] client connection to 127.0.98.1:32917 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 07:27:42.520764 14713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:27:42.521219 14712 catalog_manager.cc:4841] TS 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873): tablet 625049587e924a2ca9f7cff47e3f6165 (table test-table [id=77f00fceafc24b309273830ca3e122dd]) successfully deleted
I20260926 07:27:42.521766 15218 ts_tablet_manager.cc:1905] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:27:42.522110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 14920
I20260926 07:27:42.522501 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:42.522108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32917 (local address 127.0.0.1:44044)
0926 07:27:42.522309 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:42.522382 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260926 07:27:42.522801 14713 catalog_manager.cc:4584] TS d0245ba57cdb486a9d045eff69a77388 (127.0.98.1:32917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 625049587e924a2ca9f7cff47e3f6165: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32917: connect: Connection refused (error 111)
I20260926 07:27:42.524189 15218 ts_tablet_manager.cc:1918] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:27:42.524284 15218 log.cc:1196] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/625049587e924a2ca9f7cff47e3f6165
I20260926 07:27:42.524605 15218 ts_tablet_manager.cc:1939] T 625049587e924a2ca9f7cff47e3f6165 P 70847ca3aa2c46fcae60ad9c9332fea2: Deleting consensus metadata
I20260926 07:27:42.525653 14712 catalog_manager.cc:4841] TS 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233): tablet 625049587e924a2ca9f7cff47e3f6165 (table test-table [id=77f00fceafc24b309273830ca3e122dd]) successfully deleted
I20260926 07:27:42.529229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 15057
I20260926 07:27:42.533963 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 14695
I20260926 07:27:42.539140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42159
--webserver_interface=127.0.98.62
--webserver_port=37935
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:42.650326 15221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:42.650560 15221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:42.650599 15221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:42.654557 15221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:42.654614 15221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:42.654633 15221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:42.654649 15221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:42.654706 15221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:42.659987 15221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:42.661064 15221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:42.662096 15221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:42.670393 15226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:42.670454 15229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:42.670456 15227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:42.670917 15221 server_base.cc:1034] running on GCE node
I20260926 07:27:42.671475 15221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:42.672497 15221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:42.673655 15221 hybrid_clock.cc:648] HybridClock initialized: now 1790407662673641 us; error 26 us; skew 500 ppm
Sep 26 07:27:42 dist-test-slave-g04g krb5kdc[14147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:42.676483 15221 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:27:42.677284 15221 webserver.cc:460] Webserver started at http://127.0.98.62:37935/ using document root <none> and password file <none>
I20260926 07:27:42.677783 15221 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:42.677848 15221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:42.681555 15221 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:27:42.683591 15236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:42.684940 15221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:42.685079 15221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "eebffc3380404166a99eb1ae392d0814"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "5fcae58da9f843d3cb844e709e86df49"
server_key_iv: "cf565fce1eea765bb9b83c0726e2498d"
server_key_version: "encryptionkey@0"
I20260926 07:27:42.685626 15221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:42.698208 15221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:42.701385 15221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:42.701584 15221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:42.708393 15221 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42159
I20260926 07:27:42.708674 15288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42159 every 8 connection(s)
I20260926 07:27:42.709478 15221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:42.713610 15289 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:27:42.714895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 15221
I20260926 07:27:42.715308 15289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Bootstrap starting.
I20260926 07:27:42.715385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32917
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42159
--webserver_port=33165
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:27:42.739315 15289 log.cc:824] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:42.753171 15289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:27:42.753582 15289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814: Bootstrap complete.
I20260926 07:27:42.759168 15289 raft_consensus.cc:348] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } }
I20260926 07:27:42.759666 15289 raft_consensus.cc:729] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebffc3380404166a99eb1ae392d0814, State: Initialized, Role: FOLLOWER
I20260926 07:27:42.760215 15289 consensus_queue.cc:260] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } }
I20260926 07:27:42.760365 15289 raft_consensus.cc:388] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:42.760437 15289 raft_consensus.cc:482] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:42.760533 15289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:27:42.762956 15289 raft_consensus.cc:504] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } }
I20260926 07:27:42.763340 15289 leader_election.cc:302] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eebffc3380404166a99eb1ae392d0814; no voters:
I20260926 07:27:42.763715 15289 leader_election.cc:288] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 07:27:42.763839 15293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:27:42.764103 15293 raft_consensus.cc:686] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [term 3 LEADER]: Becoming Leader. State: Replica: eebffc3380404166a99eb1ae392d0814, State: Running, Role: LEADER
I20260926 07:27:42.764400 15293 consensus_queue.cc:237] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } }
I20260926 07:27:42.764953 15289 sys_catalog.cc:564] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:42.765888 15295 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eebffc3380404166a99eb1ae392d0814. Latest consensus state: current_term: 3 leader_uuid: "eebffc3380404166a99eb1ae392d0814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } } }
I20260926 07:27:42.766019 15295 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:42.765877 15294 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "eebffc3380404166a99eb1ae392d0814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebffc3380404166a99eb1ae392d0814" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42159 } } }
I20260926 07:27:42.766114 15294 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebffc3380404166a99eb1ae392d0814 [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:42.767587 15302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:42.771654 15302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:42.772171 15302 catalog_manager.cc:1259] Loaded cluster ID: d65d00b38d01403f8b9fccb639daeb4f
I20260926 07:27:42.772257 15302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 07:27:42.773219 15302 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1b469729ad google::LogMessage::Fail() at ??:0
@ 0x7f1b4697588a google::LogMessage::SendToLog() at ??:0
@ 0x7f1b469724de google::LogMessage::Flush() at ??:0
@ 0x7f1b46973a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1b512b1fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1b512b2d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1b512af579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1b512f3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1b48bc43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1b48bc4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1b48bc63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1b48bb5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1b4a1f96db start_thread at ??:0
@ 0x7f1b4547061f clone at ??:0
W20260926 07:27:42.874264 15291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:42.874624 15291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:42.874689 15291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:42.880769 15291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:42.880851 15291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:42.880935 15291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:27:42.888266 15291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33165
--tserver_master_addrs=127.0.98.62:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:42.889369 15291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:42.890411 15291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:42.900300 15330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:42.900346 15331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:42.900310 15333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:42.901068 15291 server_base.cc:1034] running on GCE node
I20260926 07:27:42.901501 15291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:42.902175 15291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:42.903358 15291 hybrid_clock.cc:648] HybridClock initialized: now 1790407662903328 us; error 47 us; skew 500 ppm
Sep 26 07:27:42 dist-test-slave-g04g krb5kdc[14147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:42.906184 15291 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:27:42.907086 15291 webserver.cc:460] Webserver started at http://127.0.98.1:33165/ using document root <none> and password file <none>
I20260926 07:27:42.907678 15291 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:42.907765 15291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:42.911675 15291 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 07:27:42.913930 15340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:42.915263 15291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:27:42.915388 15291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "d0245ba57cdb486a9d045eff69a77388"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "62a3d9a97e62eec2cf0ef41afd629ad7"
server_key_iv: "2ef8c00b34bfac0ea7354dc63c82db55"
server_key_version: "encryptionkey@0"
I20260926 07:27:42.915937 15291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:42.944746 15291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:42.948037 15291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:42.948241 15291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:42.948886 15291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:42.950189 15347 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:27:42.955684 15291 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:27:42.955758 15291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 07:27:42.955847 15291 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:27:42.957969 15291 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:27:42.958025 15291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 07:27:42.958388 15347 tablet_bootstrap.cc:492] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388: Bootstrap starting.
I20260926 07:27:42.969431 15291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32917
I20260926 07:27:42.969475 15454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32917 every 8 connection(s)
I20260926 07:27:42.970890 15291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:27:42.973039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 15291
I20260926 07:27:42.973505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39873
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42159
--webserver_port=33813
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:27:42.981094 15347 log.cc:824] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:42.991391 15347 tablet_bootstrap.cc:492] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:27:42.991819 15347 tablet_bootstrap.cc:492] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388: Bootstrap complete.
I20260926 07:27:42.993374 15347 ts_tablet_manager.cc:1397] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.007s
I20260926 07:27:42.996892 15347 raft_consensus.cc:348] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:42.997476 15347 raft_consensus.cc:729] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0245ba57cdb486a9d045eff69a77388, State: Initialized, Role: FOLLOWER
I20260926 07:27:42.998090 15347 consensus_queue.cc:260] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:42.999213 15347 ts_tablet_manager.cc:1428] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
W20260926 07:27:43.015201 15457 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:27:42.973277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:56913)
0926 07:27:42.975462 (+ 2185us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:42.975524 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:42.976567 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:43.014360 (+ 37793us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1939,"thread_start_us":152,"threads_started":1}
W20260926 07:27:43.015704 15455 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42159 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42159: Client connection negotiation failed: client connection to 127.0.98.62:42159: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:27:43.016420 15457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:43.016163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:34331)
0926 07:27:43.016295 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:43.016369 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 07:27:43.017047 15457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:43.016842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:53933)
0926 07:27:43.016946 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:43.016990 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:27:43.017207 15455 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:27:43.098516 15458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:43.098791 15458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:43.098884 15458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:43.102864 15458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:43.102947 15458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:43.103080 15458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:27:43.108168 15458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33813
--tserver_master_addrs=127.0.98.62:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:43.109174 15458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:43.110136 15458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:43.119894 15468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:43.119896 15466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:43.119932 15458 server_base.cc:1034] running on GCE node
W20260926 07:27:43.119894 15465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:43.120707 15458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:43.121456 15458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:43.122639 15458 hybrid_clock.cc:648] HybridClock initialized: now 1790407663122611 us; error 43 us; skew 500 ppm
Sep 26 07:27:43 dist-test-slave-g04g krb5kdc[14147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:43.125396 15458 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:27:43.126230 15458 webserver.cc:460] Webserver started at http://127.0.98.2:33813/ using document root <none> and password file <none>
I20260926 07:27:43.126791 15458 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:43.126880 15458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:43.130744 15458 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260926 07:27:43.132979 15475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:43.134171 15458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:27:43.134296 15458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd"
format_stamp: "Formatted at 2026-09-26 07:27:40 on dist-test-slave-g04g"
server_key: "5c3b638bde800f1871c56444ca7f3686"
server_key_iv: "21466aaa8ba07b1d63c650e550061af8"
server_key_version: "encryptionkey@0"
I20260926 07:27:43.134853 15458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:43.155859 15458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:43.158721 15458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:43.158934 15458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:43.159554 15458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:43.160357 15458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:43.160425 15458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:43.160496 15458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:43.160538 15458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:43.169384 15588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39873 every 8 connection(s)
I20260926 07:27:43.169359 15458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39873
I20260926 07:27:43.170827 15458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:27:43.173710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 15458
I20260926 07:27:43.174151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45233
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42159
--webserver_port=35347
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:27:43.175119 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:43.172900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:51415)
0926 07:27:43.173346 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:43.173485 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":110,"threads_started":1}
W20260926 07:27:43.175549 15589 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42159 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42159: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
I20260926 07:27:43.176234 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:43.176013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:47929)
0926 07:27:43.176134 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:43.176181 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 07:27:43.176831 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:43.176629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:48769)
0926 07:27:43.176732 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:43.176776 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 07:27:43.176978 15589 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:27:43.282611 15592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:43.282912 15592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:43.282987 15592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:43.286906 15592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:43.286999 15592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:43.287106 15592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:27:43.292143 15592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35347
--tserver_master_addrs=127.0.98.62:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:43.293121 15592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:43.294116 15592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:43.303615 15598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:43.303682 15592 server_base.cc:1034] running on GCE node
W20260926 07:27:43.303619 15597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:43.303592 15600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:43.304234 15592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:43.304895 15592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:43.306075 15592 hybrid_clock.cc:648] HybridClock initialized: now 1790407663306062 us; error 24 us; skew 500 ppm
Sep 26 07:27:43 dist-test-slave-g04g krb5kdc[14147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:43.308723 15592 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:27:43.309584 15592 webserver.cc:460] Webserver started at http://127.0.98.3:35347/ using document root <none> and password file <none>
I20260926 07:27:43.310204 15592 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:43.310297 15592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:43.313948 15592 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:43.315979 15607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:43.317201 15592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:27:43.317293 15592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "70847ca3aa2c46fcae60ad9c9332fea2"
format_stamp: "Formatted at 2026-09-26 07:27:41 on dist-test-slave-g04g"
server_key: "433f2d83d88fee334da7c7d8d67e79c5"
server_key_iv: "6f2823119d916a244c320433de8d7177"
server_key_version: "encryptionkey@0"
I20260926 07:27:43.317797 15592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:43.345186 15592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:43.347998 15592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:43.348152 15592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:43.348703 15592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:27:43.349478 15592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:27:43.349524 15592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:43.349562 15592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:27:43.349577 15592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:43.358503 15592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45233
I20260926 07:27:43.358530 15720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45233 every 8 connection(s)
I20260926 07:27:43.359520 15592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:27:43.360754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 15592
I20260926 07:27:43.363413 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:43.361646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:56079)
0926 07:27:43.362119 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:43.362224 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":137,"threads_started":1}
W20260926 07:27:43.363849 15721 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42159 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42159: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
I20260926 07:27:43.364403 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:43.364214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:55253)
0926 07:27:43.364311 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:43.364353 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 07:27:43.365016 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:43.364846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:47989)
0926 07:27:43.364933 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:43.364971 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 07:27:43.365195 15721 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:27:44.018234 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.017805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:45247)
0926 07:27:44.018069 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.018124 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":82,"threads_started":1}
I20260926 07:27:44.153478 15726 raft_consensus.cc:482] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:44.153748 15726 raft_consensus.cc:504] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:44.154973 15726 leader_election.cc:288] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873), 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233)
Sep 26 07:27:44 dist-test-slave-g04g krb5kdc[14147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 07:27:44 dist-test-slave-g04g krb5kdc[14147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 07:27:44.168514 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.155173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39873 (local address 127.0.98.1:54981)
0926 07:27:44.155295 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.155311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:44.155416 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:44.157220 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:44.157225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:44.157759 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:44.158234 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:44.158249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:44.159580 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:44.159589 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:44.160432 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:44.160441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:44.161134 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:44.161137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:44.161377 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:44.162020 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:44.162036 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:44.163898 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:44.166015 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:44.166020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:44.166033 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:44.166407 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:44.166767 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:44.166770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:44.166771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:44.166925 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:44.167546 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:44.167552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:44.167799 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 07:27:44.167979 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 07:27:44.168314 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":236,"spinlock_wait_cycles":64000}
I20260926 07:27:44.168525 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.155348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54981 (local address 127.0.98.2:39873)
0926 07:27:44.155730 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.155748 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.155800 (+ 52us) server_negotiation.cc:407] Connection header received
0926 07:27:44.156512 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.156518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.156876 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.157174 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.158396 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:44.159437 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:44.160565 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:44.161004 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:44.161185 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:44.164087 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.164091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.164101 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:44.164126 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.165876 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:44.166534 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:44.166540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:44.166545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:44.166661 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:44.167020 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:44.167024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:44.167026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:44.167427 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.167568 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.168099 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.168357 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":115,"threads_started":1}
I20260926 07:27:44.168568 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.155639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39463 (local address 127.0.98.3:45233)
0926 07:27:44.155910 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:44.155925 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:44.155963 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:27:44.156550 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:44.156555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:44.156888 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:44.157182 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:44.158411 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:44.159419 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:44.160553 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:44.161005 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:44.161184 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:44.163751 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:44.163756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:44.163767 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:44.163793 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:44.165890 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:44.166543 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:44.166548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:44.166553 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:44.166671 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:44.167024 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:44.167049 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:44.167053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:44.167427 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:44.167568 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:44.168100 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 07:27:44.168382 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
I20260926 07:27:44.168557 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.155456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45233 (local address 127.0.98.1:39463)
0926 07:27:44.155775 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.155790 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:27:44.155879 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:27:44.157219 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:27:44.157225 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:27:44.157759 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:27:44.158233 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:44.158249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:44.159575 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:44.159578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:44.160430 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:27:44.160439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:27:44.161138 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:27:44.161141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:27:44.161348 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:44.162066 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:27:44.162078 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:27:44.163525 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:27:44.166014 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:44.166020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:44.166033 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:44.166405 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:44.166769 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:27:44.166773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:27:44.166775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:27:44.166930 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:27:44.167546 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:27:44.167554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:27:44.167799 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 07:27:44.167979 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 07:27:44.168398 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":270,"spinlock_wait_cycles":30720,"thread_start_us":69,"threads_started":1}
W20260926 07:27:44.171545 15341 leader_election.cc:341] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233): Not found: Tablet not found: 625049587e924a2ca9f7cff47e3f6165
W20260926 07:27:44.172297 15341 leader_election.cc:341] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873): Not found: Tablet not found: 625049587e924a2ca9f7cff47e3f6165
I20260926 07:27:44.172374 15341 leader_election.cc:302] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0245ba57cdb486a9d045eff69a77388; no voters: 2ad0c20d79aa4f6a8e75f9b4b32938bd, 70847ca3aa2c46fcae60ad9c9332fea2
I20260926 07:27:44.172624 15726 raft_consensus.cc:2726] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:27:44.177789 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.177433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:44087)
0926 07:27:44.177653 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.177707 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260926 07:27:44.366071 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:44.365704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:59653)
0926 07:27:44.365922 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:44.365974 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":64,"threads_started":1}
I20260926 07:27:45.019723 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:45.019257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:56405)
0926 07:27:45.019557 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:45.019614 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
I20260926 07:27:45.178870 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:45.178489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:35799)
0926 07:27:45.178726 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:45.178780 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260926 07:27:45.367202 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:45.366739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:33649)
0926 07:27:45.366983 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:45.367087 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
I20260926 07:27:45.875691 15735 raft_consensus.cc:482] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:45.875818 15735 raft_consensus.cc:504] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:45.876299 15735 leader_election.cc:288] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873), 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233)
W20260926 07:27:45.877156 15341 leader_election.cc:341] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873): Not found: Tablet not found: 625049587e924a2ca9f7cff47e3f6165
W20260926 07:27:45.877285 15341 leader_election.cc:341] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233): Not found: Tablet not found: 625049587e924a2ca9f7cff47e3f6165
I20260926 07:27:45.877327 15341 leader_election.cc:302] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0245ba57cdb486a9d045eff69a77388; no voters: 2ad0c20d79aa4f6a8e75f9b4b32938bd, 70847ca3aa2c46fcae60ad9c9332fea2
I20260926 07:27:45.877480 15735 raft_consensus.cc:2726] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:27:46.020991 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:46.020557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:57015)
0926 07:27:46.020796 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:46.020876 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
I20260926 07:27:46.179943 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:46.179572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:52909)
0926 07:27:46.179803 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:46.179856 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260926 07:27:46.368301 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:46.367907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:49545)
0926 07:27:46.368166 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:46.368216 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260926 07:27:47.022250 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:47.021823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:37135)
0926 07:27:47.022094 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:47.022152 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260926 07:27:47.181073 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:47.180685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:51389)
0926 07:27:47.180916 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:47.180980 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
I20260926 07:27:47.369438 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:47.369066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:47523)
0926 07:27:47.369294 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:47.369351 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260926 07:27:48.023613 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:48.023136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:40677)
0926 07:27:48.023471 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:48.023521 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":102,"threads_started":1}
I20260926 07:27:48.182180 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:48.181801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:56523)
0926 07:27:48.182039 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:48.182091 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
I20260926 07:27:48.245440 15744 raft_consensus.cc:482] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:48.245563 15744 raft_consensus.cc:504] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:48.245898 15744 leader_election.cc:288] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873), 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233)
W20260926 07:27:48.246809 15341 leader_election.cc:341] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233): Not found: Tablet not found: 625049587e924a2ca9f7cff47e3f6165
W20260926 07:27:48.246969 15341 leader_election.cc:341] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873): Not found: Tablet not found: 625049587e924a2ca9f7cff47e3f6165
I20260926 07:27:48.247032 15341 leader_election.cc:302] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0245ba57cdb486a9d045eff69a77388; no voters: 2ad0c20d79aa4f6a8e75f9b4b32938bd, 70847ca3aa2c46fcae60ad9c9332fea2
I20260926 07:27:48.247223 15744 raft_consensus.cc:2726] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:27:48.370587 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:48.370174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:48011)
0926 07:27:48.370441 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:48.370498 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
I20260926 07:27:49.024794 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:49.024371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:60151)
0926 07:27:49.024643 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:49.024698 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20260926 07:27:49.183360 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:49.182913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:58881)
0926 07:27:49.183194 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:49.183247 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":71,"threads_started":1}
I20260926 07:27:49.371834 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:49.371409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:40319)
0926 07:27:49.371676 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:49.371737 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":67,"threads_started":1}
I20260926 07:27:50.026026 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:50.025601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:48205)
0926 07:27:50.025875 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:50.025931 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":115,"threads_started":1}
I20260926 07:27:50.184499 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:50.184066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:45809)
0926 07:27:50.184352 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:50.184406 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":68,"threads_started":1}
I20260926 07:27:50.372967 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:50.372527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:33359)
0926 07:27:50.372823 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:50.372878 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":71,"threads_started":1}
I20260926 07:27:51.027247 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:51.026793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:43339)
0926 07:27:51.027084 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:51.027152 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":85,"threads_started":1}
I20260926 07:27:51.185570 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:51.185165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:40051)
0926 07:27:51.185419 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:51.185472 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260926 07:27:51.374091 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:51.373671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:53445)
0926 07:27:51.373936 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:51.374000 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
I20260926 07:27:52.028292 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:52.027921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:35191)
0926 07:27:52.028147 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:52.028198 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260926 07:27:52.152334 15756 raft_consensus.cc:482] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:52.152565 15756 raft_consensus.cc:504] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:52.152987 15756 leader_election.cc:288] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873), 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233)
W20260926 07:27:52.153857 15341 leader_election.cc:341] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233): Not found: Tablet not found: 625049587e924a2ca9f7cff47e3f6165
W20260926 07:27:52.153964 15341 leader_election.cc:341] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873): Not found: Tablet not found: 625049587e924a2ca9f7cff47e3f6165
I20260926 07:27:52.154034 15341 leader_election.cc:302] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0245ba57cdb486a9d045eff69a77388; no voters: 2ad0c20d79aa4f6a8e75f9b4b32938bd, 70847ca3aa2c46fcae60ad9c9332fea2
I20260926 07:27:52.154237 15756 raft_consensus.cc:2726] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:27:52.186784 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:52.186338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:43573)
0926 07:27:52.186628 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:52.186687 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":76,"threads_started":1}
I20260926 07:27:52.375241 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:52.374792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:38843)
0926 07:27:52.375100 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:52.375151 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":94,"threads_started":1}
I20260926 07:27:53.029525 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:53.029133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:50617)
0926 07:27:53.029372 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:53.029426 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260926 07:27:53.187901 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:53.187506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:58717)
0926 07:27:53.187749 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:53.187801 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
I20260926 07:27:53.376431 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:53.376019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:50661)
0926 07:27:53.376278 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:53.376333 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260926 07:27:54.030691 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:54.030283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:39281)
0926 07:27:54.030544 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:54.030598 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":95,"threads_started":1}
I20260926 07:27:54.189039 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:54.188631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:33459)
0926 07:27:54.188886 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:54.188943 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
I20260926 07:27:54.377624 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:54.377176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:46409)
0926 07:27:54.377445 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:54.377502 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":67,"threads_started":1}
I20260926 07:27:55.031919 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:55.031496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:53473)
0926 07:27:55.031765 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:55.031823 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
I20260926 07:27:55.190207 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:55.189817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:55597)
0926 07:27:55.190051 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:55.190104 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260926 07:27:55.229787 15767 raft_consensus.cc:482] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:27:55.229909 15767 raft_consensus.cc:504] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ad0c20d79aa4f6a8e75f9b4b32938bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } } peers { permanent_uuid: "70847ca3aa2c46fcae60ad9c9332fea2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45233 } } peers { permanent_uuid: "d0245ba57cdb486a9d045eff69a77388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32917 } }
I20260926 07:27:55.230382 15767 leader_election.cc:288] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873), 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233)
W20260926 07:27:55.231216 15341 leader_election.cc:341] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70847ca3aa2c46fcae60ad9c9332fea2 (127.0.98.3:45233): Not found: Tablet not found: 625049587e924a2ca9f7cff47e3f6165
W20260926 07:27:55.231365 15341 leader_election.cc:341] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ad0c20d79aa4f6a8e75f9b4b32938bd (127.0.98.2:39873): Not found: Tablet not found: 625049587e924a2ca9f7cff47e3f6165
I20260926 07:27:55.231429 15341 leader_election.cc:302] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0245ba57cdb486a9d045eff69a77388; no voters: 2ad0c20d79aa4f6a8e75f9b4b32938bd, 70847ca3aa2c46fcae60ad9c9332fea2
I20260926 07:27:55.231588 15767 raft_consensus.cc:2726] T 625049587e924a2ca9f7cff47e3f6165 P d0245ba57cdb486a9d045eff69a77388 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:27:55.378890 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:55.378500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:56661)
0926 07:27:55.378746 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:55.378802 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
I20260926 07:27:56.033141 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.032764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:34083)
0926 07:27:56.032999 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:56.033048 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
I20260926 07:27:56.191380 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.190952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:36333)
0926 07:27:56.191229 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:56.191285 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":103,"threads_started":1}
I20260926 07:27:56.380030 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:56.379641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:45873)
0926 07:27:56.379869 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:56.379926 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260926 07:27:57.034272 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.033860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:33659)
0926 07:27:57.034122 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.034177 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260926 07:27:57.192483 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.192071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:38685)
0926 07:27:57.192334 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.192387 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":71,"threads_started":1}
I20260926 07:27:57.381187 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:57.380748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.3:43859)
0926 07:27:57.381040 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:57.381094 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
I20260926 07:27:58.035612 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:58.035237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.1:41077)
0926 07:27:58.035462 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:58.035519 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260926 07:27:58.193735 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:58.193351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42159 (local address 127.0.98.2:42627)
0926 07:27:58.193587 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 07:27:58.193641 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260926 07:27:58.369232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 15291
I20260926 07:27:58.375000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 15458
I20260926 07:27:58.380707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 15592
2026-09-26T07:27:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19204 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:27:58 dist-test-slave-g04g krb5kdc[15779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:27:58 dist-test-slave-g04g krb5kdc[15779](info): set up 2 sockets
Sep 26 07:27:58 dist-test-slave-g04g krb5kdc[15779](info): commencing operation
krb5kdc: starting...
W20260926 07:27:59.427327 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:27:59 dist-test-slave-g04g krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:27:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:27:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:27:59.545866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46307
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:59.654351 15792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:59.654663 15792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:59.654744 15792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:59.658879 15792 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:27:59.658957 15792 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:27:59.658984 15792 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:27:59.659015 15792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:27:59.659070 15792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:59.659133 15792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:59.659165 15792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:27:59.659197 15792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:27:59.664413 15792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:59.665445 15792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:59.666458 15792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:27:59.675302 15798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:59.675307 15797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:59.675354 15792 server_base.cc:1034] running on GCE node
W20260926 07:27:59.675335 15800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:59.676180 15792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:59.677239 15792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:59.678424 15792 hybrid_clock.cc:648] HybridClock initialized: now 1790407679678397 us; error 43 us; skew 500 ppm
Sep 26 07:27:59 dist-test-slave-g04g krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:59.681121 15792 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:27:59.682014 15792 webserver.cc:460] Webserver started at http://127.0.98.62:36879/ using document root <none> and password file <none>
I20260926 07:27:59.682583 15792 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:59.682642 15792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:59.682865 15792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:59.685115 15792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "c453ab1ec5a24671816da058aeee41ea"
format_stamp: "Formatted at 2026-09-26 07:27:59 on dist-test-slave-g04g"
server_key: "09a47ae2e79fa689f38f862af565e839"
server_key_iv: "999d47b10e7af5cba7286a7a1b795f07"
server_key_version: "encryptionkey@0"
I20260926 07:27:59.685708 15792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "c453ab1ec5a24671816da058aeee41ea"
format_stamp: "Formatted at 2026-09-26 07:27:59 on dist-test-slave-g04g"
server_key: "09a47ae2e79fa689f38f862af565e839"
server_key_iv: "999d47b10e7af5cba7286a7a1b795f07"
server_key_version: "encryptionkey@0"
I20260926 07:27:59.689359 15792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260926 07:27:59.691838 15807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:59.693049 15792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 07:27:59.693174 15792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "c453ab1ec5a24671816da058aeee41ea"
format_stamp: "Formatted at 2026-09-26 07:27:59 on dist-test-slave-g04g"
server_key: "09a47ae2e79fa689f38f862af565e839"
server_key_iv: "999d47b10e7af5cba7286a7a1b795f07"
server_key_version: "encryptionkey@0"
I20260926 07:27:59.693277 15792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:27:59.733485 15792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:27:59.736558 15792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:27:59.736765 15792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:27:59.744043 15792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46307
I20260926 07:27:59.744330 15859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46307 every 8 connection(s)
I20260926 07:27:59.745070 15792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:27:59.747884 15860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:27:59.752216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 15792
I20260926 07:27:59.752341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:27:59.752635 392 external_mini_cluster.cc:1428] Setting key 238e50c8cdb58ca3d9a5ac00df4fc213
I20260926 07:27:59.754002 15860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Bootstrap starting.
I20260926 07:27:59.756583 15860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Neither blocks nor log segments found. Creating new log.
I20260926 07:27:59.757359 15860 log.cc:824] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Log is configured to *not* fsync() on all Append() calls
I20260926 07:27:59.759179 15860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: No bootstrap required, opened a new log
Sep 26 07:27:59 dist-test-slave-g04g krb5kdc[15779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:27:59.762820 15860 raft_consensus.cc:348] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260926 07:27:59.763106 15860 raft_consensus.cc:374] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:27:59.763191 15860 raft_consensus.cc:729] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c453ab1ec5a24671816da058aeee41ea, State: Initialized, Role: FOLLOWER
I20260926 07:27:59.763636 15860 consensus_queue.cc:260] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260926 07:27:59.763792 15860 raft_consensus.cc:388] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:27:59.763861 15860 raft_consensus.cc:482] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:27:59.763954 15860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:27:59.764776 15860 raft_consensus.cc:504] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260926 07:27:59.765126 15860 leader_election.cc:302] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c453ab1ec5a24671816da058aeee41ea; no voters:
I20260926 07:27:59.765450 15860 leader_election.cc:288] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:27:59.765568 15865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:27:59.765826 15865 raft_consensus.cc:686] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 1 LEADER]: Becoming Leader. State: Replica: c453ab1ec5a24671816da058aeee41ea, State: Running, Role: LEADER
I20260926 07:27:59.766184 15865 consensus_queue.cc:237] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260926 07:27:59.766685 15860 sys_catalog.cc:564] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:27:59.767364 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:27:59.754025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.0.98.62:46307)
0926 07:27:59.754487 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 07:27:59.754504 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:27:59.754539 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:27:59.755243 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:27:59.755262 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:27:59.755622 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:27:59.755979 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:27:59.756873 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.757908 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.758759 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:27:59.759315 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:27:59.759497 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:27:59.761897 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:27:59.761902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:27:59.761914 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:27:59.761940 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:27:59.763830 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.764316 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.764324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.764334 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.764440 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:27:59.764732 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:27:59.764736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:27:59.764738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:27:59.765203 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:27:59.765402 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 07:27:59.766098 (+ 696us) server_negotiation.cc:299] Negotiation successful
0926 07:27:59.766375 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":135,"threads_started":1}
I20260926 07:27:59.767800 15866 sys_catalog.cc:455] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c453ab1ec5a24671816da058aeee41ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } } }
I20260926 07:27:59.767930 15866 sys_catalog.cc:458] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:59.768328 15874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:27:59.768671 15867 sys_catalog.cc:455] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader c453ab1ec5a24671816da058aeee41ea. Latest consensus state: current_term: 1 leader_uuid: "c453ab1ec5a24671816da058aeee41ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } } }
I20260926 07:27:59.768831 15867 sys_catalog.cc:458] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: This master's current role is: LEADER
I20260926 07:27:59.771306 15874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:27:59.776842 15874 catalog_manager.cc:1347] Generated new cluster ID: 96c29ba8e487444aaadf6c3c322a6a95
I20260926 07:27:59.776906 15874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:27:59.790763 15874 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:27:59.792034 15874 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:27:59.802644 15874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Generated new TSK 0
I20260926 07:27:59.803486 15874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:27:59.830910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46307
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:27:59.941046 15886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:27:59.941304 15886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:27:59.941349 15886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:27:59.947016 15886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:27:59.947132 15886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:27:59.947219 15886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:27:59.953207 15886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:27:59.954298 15886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:27:59.955399 15886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:27:59.965003 15886 server_base.cc:1034] running on GCE node
W20260926 07:27:59.964982 15891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:59.965101 15892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:27:59.964975 15894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:27:59.965587 15886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:27:59.966214 15886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:27:59.967401 15886 hybrid_clock.cc:648] HybridClock initialized: now 1790407679967387 us; error 25 us; skew 500 ppm
Sep 26 07:27:59 dist-test-slave-g04g krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:27:59.970082 15886 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:27:59.970954 15886 webserver.cc:460] Webserver started at http://127.0.98.1:37575/ using document root <none> and password file <none>
I20260926 07:27:59.971539 15886 fs_manager.cc:359] Metadata directory not provided
I20260926 07:27:59.971628 15886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:27:59.971863 15886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:27:59.974059 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "055a3aea67e34bafa5dbb2a0a347e50a"
format_stamp: "Formatted at 2026-09-26 07:27:59 on dist-test-slave-g04g"
server_key: "cb0cadcecf90173e0630443a57255e0b"
server_key_iv: "6157525abbd170438f6db96de47ef4ea"
server_key_version: "encryptionkey@0"
I20260926 07:27:59.974659 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "055a3aea67e34bafa5dbb2a0a347e50a"
format_stamp: "Formatted at 2026-09-26 07:27:59 on dist-test-slave-g04g"
server_key: "cb0cadcecf90173e0630443a57255e0b"
server_key_iv: "6157525abbd170438f6db96de47ef4ea"
server_key_version: "encryptionkey@0"
I20260926 07:27:59.978514 15886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 07:27:59.981604 15901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:27:59.983258 15886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:27:59.983395 15886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "055a3aea67e34bafa5dbb2a0a347e50a"
format_stamp: "Formatted at 2026-09-26 07:27:59 on dist-test-slave-g04g"
server_key: "cb0cadcecf90173e0630443a57255e0b"
server_key_iv: "6157525abbd170438f6db96de47ef4ea"
server_key_version: "encryptionkey@0"
I20260926 07:27:59.983529 15886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:00.008531 15886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:00.011920 15886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:00.012073 15886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:00.012595 15886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:00.013357 15886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:00.013405 15886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:00.013444 15886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:00.013459 15886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:00.022316 15886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46775
I20260926 07:28:00.022349 16014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46775 every 8 connection(s)
I20260926 07:28:00.023272 15886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:28:00.027163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 15886
I20260926 07:28:00.027325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:28:00.027659 392 external_mini_cluster.cc:1428] Setting key e12687e4e5ba3d142c1a6e107d0f7421
Sep 26 07:28:00 dist-test-slave-g04g krb5kdc[15779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 07:28:00.037994 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.025146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54085 (local address 127.0.98.62:46307)
0926 07:28:00.025297 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:00.025301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:00.026269 (+ 968us) server_negotiation.cc:407] Connection header received
0926 07:28:00.027294 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:00.027299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:00.027364 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:00.027466 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:00.028898 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.029559 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.030824 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.031143 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.031273 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.034013 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:00.034020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:00.034026 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:00.034064 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:00.035968 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:00.036470 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:00.036477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:00.036480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:00.036576 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:00.036894 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:00.036900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:00.036902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:00.037147 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:00.037261 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:00.037696 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 07:28:00.037808 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:28:00.038880 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.025450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:54085)
0926 07:28:00.026104 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:00.026147 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:00.027003 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:00.027627 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:00.027634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:00.028134 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:00.028695 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.028710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.029703 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.029707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.030683 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.030693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.031308 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.031313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.031496 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.032111 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:00.032128 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:00.033836 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:00.036086 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:00.036090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:00.036102 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:00.036358 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:00.036672 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:00.036674 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:00.036676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:00.036801 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:00.037251 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:00.037256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:00.037484 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 07:28:00.037641 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 07:28:00.037983 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 07:28:00.040225 16015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46307
I20260926 07:28:00.040481 16015 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:00.041082 16015 heartbeater.cc:511] Master 127.0.98.62:46307 requested a full tablet report, sending...
I20260926 07:28:00.042768 15824 ts_manager.cc:194] Registered new tserver with Master: 055a3aea67e34bafa5dbb2a0a347e50a (127.0.98.1:46775)
I20260926 07:28:00.044480 15824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:28:00.057983 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46307
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:00.172847 16020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:00.173094 16020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:00.173177 16020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:00.177193 16020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:00.177254 16020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:00.177349 16020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:28:00.183204 16020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:00.184258 16020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:00.185329 16020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:00.195009 16025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:00.195017 16026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:00.195024 16028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:00.195317 16020 server_base.cc:1034] running on GCE node
I20260926 07:28:00.195765 16020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:00.196374 16020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:00.197559 16020 hybrid_clock.cc:648] HybridClock initialized: now 1790407680197538 us; error 33 us; skew 500 ppm
Sep 26 07:28:00 dist-test-slave-g04g krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:00.218840 16020 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:28:00.219841 16020 webserver.cc:460] Webserver started at http://127.0.98.2:45313/ using document root <none> and password file <none>
I20260926 07:28:00.220471 16020 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:00.220580 16020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:00.220811 16020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:00.222899 16020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "2ce29db40e9a47169f9ec63fccd90d1f"
format_stamp: "Formatted at 2026-09-26 07:28:00 on dist-test-slave-g04g"
server_key: "c94fe220c246746b3ad964b665618665"
server_key_iv: "f97fd1d65c0d1152fce970dff75bf1ec"
server_key_version: "encryptionkey@0"
I20260926 07:28:00.223502 16020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2ce29db40e9a47169f9ec63fccd90d1f"
format_stamp: "Formatted at 2026-09-26 07:28:00 on dist-test-slave-g04g"
server_key: "c94fe220c246746b3ad964b665618665"
server_key_iv: "f97fd1d65c0d1152fce970dff75bf1ec"
server_key_version: "encryptionkey@0"
I20260926 07:28:00.227166 16020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:28:00.229552 16035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:00.230732 16020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:00.230872 16020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "2ce29db40e9a47169f9ec63fccd90d1f"
format_stamp: "Formatted at 2026-09-26 07:28:00 on dist-test-slave-g04g"
server_key: "c94fe220c246746b3ad964b665618665"
server_key_iv: "f97fd1d65c0d1152fce970dff75bf1ec"
server_key_version: "encryptionkey@0"
I20260926 07:28:00.231000 16020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:00.249557 16020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:00.252466 16020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:00.252679 16020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:00.253266 16020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:00.254318 16020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:00.254412 16020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:00.254479 16020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:00.254534 16020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:00.264837 16148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38273 every 8 connection(s)
I20260926 07:28:00.264988 16020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38273
I20260926 07:28:00.266587 16020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:28:00.274852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 16020
I20260926 07:28:00.274982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:28:00.275436 392 external_mini_cluster.cc:1428] Setting key e365c80ae86c5e4110f34e9c4f4bac4f
Sep 26 07:28:00 dist-test-slave-g04g krb5kdc[15779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:28:00.285159 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.269309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47595 (local address 127.0.98.62:46307)
0926 07:28:00.269462 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:00.269469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:00.270456 (+ 987us) server_negotiation.cc:407] Connection header received
0926 07:28:00.271896 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:00.271929 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:00.271987 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:00.272071 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:00.274147 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.274981 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.276684 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.276972 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.277068 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.280349 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:00.280353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:00.280356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:00.280387 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:00.282313 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:00.282962 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:00.282966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:00.282968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:00.283114 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:00.283609 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:00.283616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:00.283620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:00.283864 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:00.283941 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:00.284831 (+ 890us) server_negotiation.cc:299] Negotiation successful
0926 07:28:00.284964 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:28:00.286612 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.269419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:47595)
0926 07:28:00.270295 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:00.270350 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:00.271473 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:00.272492 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:00.272501 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:00.273122 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:00.273930 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.273947 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.275156 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.275160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.276571 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.276581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.277265 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.277269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.277460 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.278154 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:00.278179 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:00.280141 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:00.282442 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:00.282448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:00.282465 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:00.282855 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:00.283264 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:00.283270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:00.283275 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:00.283484 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:00.284028 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:00.284035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:00.284476 (+ 441us) client_negotiation.cc:769] Sending connection context
0926 07:28:00.284770 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 07:28:00.285106 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":154,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 07:28:00.288400 16149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46307
I20260926 07:28:00.288691 16149 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:00.289288 16149 heartbeater.cc:511] Master 127.0.98.62:46307 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 07:28:00.290475 15824 ts_manager.cc:194] Registered new tserver with Master: 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273)
I20260926 07:28:00.291333 15824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:28:00.311368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46307
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:00.421726 16154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:00.422029 16154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:00.422102 16154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:00.426092 16154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:00.426190 16154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:00.426288 16154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:28:00.431478 16154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:00.432530 16154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:00.433766 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:28:00.443363 16154 server_base.cc:1034] running on GCE node
W20260926 07:28:00.443318 16160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:00.443306 16162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:00.443334 16159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:00.443955 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:00.444566 16154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:00.445775 16154 hybrid_clock.cc:648] HybridClock initialized: now 1790407680445746 us; error 49 us; skew 500 ppm
Sep 26 07:28:00 dist-test-slave-g04g krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:00.448490 16154 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:28:00.449249 16154 webserver.cc:460] Webserver started at http://127.0.98.3:36701/ using document root <none> and password file <none>
I20260926 07:28:00.449749 16154 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:00.449805 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:00.449955 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:00.451980 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "4901232023b24e9c91576f95abce51a3"
format_stamp: "Formatted at 2026-09-26 07:28:00 on dist-test-slave-g04g"
server_key: "553fdaef3b4a6c6d48a94488424c7fb4"
server_key_iv: "10567e9a6a584f375670ddc71da3e412"
server_key_version: "encryptionkey@0"
I20260926 07:28:00.452467 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4901232023b24e9c91576f95abce51a3"
format_stamp: "Formatted at 2026-09-26 07:28:00 on dist-test-slave-g04g"
server_key: "553fdaef3b4a6c6d48a94488424c7fb4"
server_key_iv: "10567e9a6a584f375670ddc71da3e412"
server_key_version: "encryptionkey@0"
I20260926 07:28:00.456014 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:28:00.458343 16169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:00.459597 16154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:00.459688 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "4901232023b24e9c91576f95abce51a3"
format_stamp: "Formatted at 2026-09-26 07:28:00 on dist-test-slave-g04g"
server_key: "553fdaef3b4a6c6d48a94488424c7fb4"
server_key_iv: "10567e9a6a584f375670ddc71da3e412"
server_key_version: "encryptionkey@0"
I20260926 07:28:00.459806 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:00.470554 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:00.473273 16154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:00.473412 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:00.473887 16154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:00.474617 16154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:00.474660 16154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:00.474699 16154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:00.474713 16154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:00.482931 16154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33189
I20260926 07:28:00.483016 16282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33189 every 8 connection(s)
I20260926 07:28:00.483852 16154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:28:00.487600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 16154
I20260926 07:28:00.487735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:28:00.488019 392 external_mini_cluster.cc:1428] Setting key 7f15f0c51160464762836ea26866559e
Sep 26 07:28:00 dist-test-slave-g04g krb5kdc[15779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:28:00.497421 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.485901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49655 (local address 127.0.98.62:46307)
0926 07:28:00.486047 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:00.486051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:00.487026 (+ 975us) server_negotiation.cc:407] Connection header received
0926 07:28:00.488125 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:00.488128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:00.488188 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:00.488312 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:00.489597 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.490337 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.491537 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.491740 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.491822 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.494146 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:00.494149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:00.494151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:00.494175 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:00.495510 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:00.495992 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:00.495995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:00.495996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:00.496066 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:00.496355 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:00.496357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:00.496358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:00.496516 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:00.496590 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:00.497176 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 07:28:00.497289 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:28:00.498385 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.486149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:49655)
0926 07:28:00.486864 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:00.486906 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:00.487911 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:00.488484 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:00.488494 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:00.488886 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:00.489449 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.489469 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.490458 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.490461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.491433 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.491442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.491866 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.491869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.492010 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.492557 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:00.492572 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:00.493990 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:00.495610 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:00.495614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:00.495625 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:00.495900 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:00.496145 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:00.496147 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:00.496149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:00.496274 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:00.496602 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:00.496607 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:00.496861 (+ 254us) client_negotiation.cc:769] Sending connection context
0926 07:28:00.497099 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 07:28:00.497396 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":125,"threads_started":1}
I20260926 07:28:00.499514 16283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46307
I20260926 07:28:00.499766 16283 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:00.500263 16283 heartbeater.cc:511] Master 127.0.98.62:46307 requested a full tablet report, sending...
I20260926 07:28:00.501318 15824 ts_manager.cc:194] Registered new tserver with Master: 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189)
I20260926 07:28:00.502071 15824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49655
I20260926 07:28:00.502147 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:28:00.512001 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.504504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.0.98.62:46307)
0926 07:28:00.504734 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:00.504738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:00.504864 (+ 126us) server_negotiation.cc:407] Connection header received
0926 07:28:00.504956 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:00.504959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:00.504997 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:00.505073 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:00.505735 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.506272 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.507122 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.507379 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.507468 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.508744 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:00.508747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:00.508749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:00.508772 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:00.510381 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:00.510737 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:00.510740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:00.510741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:00.510830 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:00.511112 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:00.511117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:00.511120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:00.511340 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:00.511450 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:00.511663 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 07:28:00.511794 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260926 07:28:00.515102 15824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:28:00.518276 15824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:28:00.533170 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.527021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.0.98.3:33189)
0926 07:28:00.529114 (+ 2093us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:00.529128 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:00.529149 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:28:00.529219 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:00.529227 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:00.529393 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:00.529549 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:00.529770 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.530683 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.531895 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.532457 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.532561 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.532707 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:00.532928 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 07:28:00.533024 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1974,"thread_start_us":116,"threads_started":1}
I20260926 07:28:00.533162 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.526899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33189 (local address 127.0.0.1:52906)
0926 07:28:00.527724 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:00.527773 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:00.527921 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:00.529529 (+ 1608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:00.529532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:00.529548 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:00.529650 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.529657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.530845 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.530848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.531759 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.531768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.532571 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.532583 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.532711 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.532756 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 07:28:00.532877 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 07:28:00.532989 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"spinlock_wait_cycles":12928,"thread_start_us":85,"threads_started":1}
I20260926 07:28:00.533946 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.528106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46775 (local address 127.0.0.1:53408)
0926 07:28:00.528375 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:00.528391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:00.528498 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:00.529052 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:00.529055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:00.529072 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:00.529176 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.529183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.530610 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.530614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.532355 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.532367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.533500 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.533504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.533612 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.533639 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 07:28:00.533731 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:28:00.533808 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":191,"thread_start_us":97,"threads_started":1}
I20260926 07:28:00.533999 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.527318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.0.98.1:46775)
0926 07:28:00.528375 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:00.528393 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:00.528484 (+ 91us) server_negotiation.cc:407] Connection header received
0926 07:28:00.528657 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:00.528663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:00.528876 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:00.529042 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:00.529309 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.530436 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.532519 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.533351 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.533503 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.533652 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:00.533758 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 07:28:00.533849 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":137,"threads_started":1}
I20260926 07:28:00.534555 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.527350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38273 (local address 127.0.0.1:47664)
0926 07:28:00.527754 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:00.527775 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:00.527922 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:00.528843 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:00.528849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:00.528874 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:00.529036 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.529045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.530639 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.530642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.532355 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.532367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.534152 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.534156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.534255 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.534277 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:28:00.534358 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:28:00.534422 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":138,"spinlock_wait_cycles":7680,"thread_start_us":106,"threads_started":1}
I20260926 07:28:00.535145 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.527849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.0.98.2:38273)
0926 07:28:00.528448 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:00.528457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:00.528477 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:28:00.528538 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:00.528542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:00.528695 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:00.528823 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:00.529192 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.530467 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.532506 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.533345 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.534734 (+ 1389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.534860 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:00.534901 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 07:28:00.534967 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":118,"threads_started":1}
I20260926 07:28:00.536878 15949 tablet_service.cc:1467] Processing CreateTablet for tablet f05ca8a0ae904c3e8752352b8a19cc6b (DEFAULT_TABLE table=test-table [id=650feac167a84063a72373a489569d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:00.538113 15949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05ca8a0ae904c3e8752352b8a19cc6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:00.537782 16083 tablet_service.cc:1467] Processing CreateTablet for tablet f05ca8a0ae904c3e8752352b8a19cc6b (DEFAULT_TABLE table=test-table [id=650feac167a84063a72373a489569d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:00.536885 16217 tablet_service.cc:1467] Processing CreateTablet for tablet f05ca8a0ae904c3e8752352b8a19cc6b (DEFAULT_TABLE table=test-table [id=650feac167a84063a72373a489569d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:00.538851 16083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05ca8a0ae904c3e8752352b8a19cc6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:00.538952 16217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05ca8a0ae904c3e8752352b8a19cc6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:00.544203 16300 tablet_bootstrap.cc:492] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Bootstrap starting.
I20260926 07:28:00.545904 16301 tablet_bootstrap.cc:492] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3: Bootstrap starting.
I20260926 07:28:00.546442 16302 tablet_bootstrap.cc:492] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: Bootstrap starting.
I20260926 07:28:00.546518 16300 tablet_bootstrap.cc:654] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:00.547403 16300 log.cc:824] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:00.548990 16301 tablet_bootstrap.cc:654] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:00.549013 16302 tablet_bootstrap.cc:654] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:00.549613 16300 tablet_bootstrap.cc:492] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: No bootstrap required, opened a new log
I20260926 07:28:00.549806 16300 ts_tablet_manager.cc:1397] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:28:00.549790 16302 log.cc:824] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:00.550112 16301 log.cc:824] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:00.551786 16302 tablet_bootstrap.cc:492] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: No bootstrap required, opened a new log
I20260926 07:28:00.551990 16302 ts_tablet_manager.cc:1397] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:28:00.552146 16301 tablet_bootstrap.cc:492] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3: No bootstrap required, opened a new log
I20260926 07:28:00.552341 16301 ts_tablet_manager.cc:1397] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 07:28:00.553681 16300 raft_consensus.cc:348] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } } peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } }
I20260926 07:28:00.553925 16300 raft_consensus.cc:374] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:00.553996 16300 raft_consensus.cc:729] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055a3aea67e34bafa5dbb2a0a347e50a, State: Initialized, Role: FOLLOWER
I20260926 07:28:00.554522 16300 consensus_queue.cc:260] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } } peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } }
I20260926 07:28:00.555585 16300 ts_tablet_manager.cc:1428] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:28:00.555968 16015 heartbeater.cc:503] Master 127.0.98.62:46307 was elected leader, sending a full tablet report...
I20260926 07:28:00.556113 16302 raft_consensus.cc:348] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } } peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } }
I20260926 07:28:00.556105 16301 raft_consensus.cc:348] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } } peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } }
I20260926 07:28:00.556363 16301 raft_consensus.cc:374] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:00.556363 16302 raft_consensus.cc:374] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:00.556416 16301 raft_consensus.cc:729] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4901232023b24e9c91576f95abce51a3, State: Initialized, Role: FOLLOWER
I20260926 07:28:00.556416 16302 raft_consensus.cc:729] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce29db40e9a47169f9ec63fccd90d1f, State: Initialized, Role: FOLLOWER
I20260926 07:28:00.556841 16301 consensus_queue.cc:260] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } } peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } }
I20260926 07:28:00.556841 16302 consensus_queue.cc:260] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } } peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } }
I20260926 07:28:00.557819 16301 ts_tablet_manager.cc:1428] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260926 07:28:00.557819 16302 ts_tablet_manager.cc:1428] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:28:00.557883 16149 heartbeater.cc:503] Master 127.0.98.62:46307 was elected leader, sending a full tablet report...
I20260926 07:28:00.558888 16283 heartbeater.cc:503] Master 127.0.98.62:46307 was elected leader, sending a full tablet report...
I20260926 07:28:00.706208 16306 raft_consensus.cc:482] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:00.706410 16306 raft_consensus.cc:504] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } } peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } }
I20260926 07:28:00.707794 16306 leader_election.cc:288] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189), 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273)
I20260926 07:28:00.712085 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.708099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38273 (local address 127.0.98.1:39671)
0926 07:28:00.708497 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:00.708511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:00.708632 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:00.708896 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:00.708898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:00.708914 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:00.709034 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.709046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.709771 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.709776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.710885 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.710895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.711626 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.711629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.711724 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.711750 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:28:00.711851 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 07:28:00.711922 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":142,"thread_start_us":149,"threads_started":1}
I20260926 07:28:00.712059 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.708394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39671 (local address 127.0.98.2:38273)
0926 07:28:00.708531 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:00.708538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:00.708580 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:28:00.708730 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:00.708734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:00.708794 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:00.708888 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:00.709156 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.709656 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.711062 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.711490 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.711624 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.711686 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:00.711876 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 07:28:00.711946 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:28:00.712196 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.708196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57623 (local address 127.0.98.3:33189)
0926 07:28:00.708352 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:00.708357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:00.708647 (+ 290us) server_negotiation.cc:407] Connection header received
0926 07:28:00.708818 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:00.708822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:00.708882 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:00.708982 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:00.709277 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.709784 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.711071 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.711504 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.711656 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.711727 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:00.711951 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 07:28:00.712027 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:28:00.712610 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.708120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33189 (local address 127.0.98.1:57623)
0926 07:28:00.708560 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:00.708577 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:00.708689 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:00.708991 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:00.708994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:00.709010 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:00.709158 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.709163 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.709915 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.709920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.710881 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:00.710891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.711653 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.711657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:00.711769 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.711795 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 07:28:00.712358 (+ 563us) client_negotiation.cc:240] Negotiation successful
0926 07:28:00.712448 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":64,"thread_start_us":133,"threads_started":1}
I20260926 07:28:00.712970 16103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05ca8a0ae904c3e8752352b8a19cc6b" candidate_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" is_pre_election: true
I20260926 07:28:00.713300 16103 raft_consensus.cc:2454] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 055a3aea67e34bafa5dbb2a0a347e50a in term 0.
I20260926 07:28:00.713318 16237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05ca8a0ae904c3e8752352b8a19cc6b" candidate_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4901232023b24e9c91576f95abce51a3" is_pre_election: true
I20260926 07:28:00.713618 16237 raft_consensus.cc:2454] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 055a3aea67e34bafa5dbb2a0a347e50a in term 0.
I20260926 07:28:00.713869 15905 leader_election.cc:302] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055a3aea67e34bafa5dbb2a0a347e50a, 2ce29db40e9a47169f9ec63fccd90d1f; no voters:
I20260926 07:28:00.714175 16306 raft_consensus.cc:2781] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:28:00.714249 16306 raft_consensus.cc:482] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:28:00.714282 16306 raft_consensus.cc:3037] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:00.715437 16306 raft_consensus.cc:504] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } } peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } }
I20260926 07:28:00.715757 16306 leader_election.cc:288] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 1 election: Requested vote from peers 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189), 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273)
I20260926 07:28:00.716195 16237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05ca8a0ae904c3e8752352b8a19cc6b" candidate_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4901232023b24e9c91576f95abce51a3"
I20260926 07:28:00.716351 16237 raft_consensus.cc:3037] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:00.716354 16103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05ca8a0ae904c3e8752352b8a19cc6b" candidate_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ce29db40e9a47169f9ec63fccd90d1f"
I20260926 07:28:00.716518 16103 raft_consensus.cc:3037] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:00.717432 16237 raft_consensus.cc:2454] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 055a3aea67e34bafa5dbb2a0a347e50a in term 1.
I20260926 07:28:00.717633 16103 raft_consensus.cc:2454] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 055a3aea67e34bafa5dbb2a0a347e50a in term 1.
I20260926 07:28:00.717810 15902 leader_election.cc:302] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055a3aea67e34bafa5dbb2a0a347e50a, 4901232023b24e9c91576f95abce51a3; no voters:
I20260926 07:28:00.718011 16306 raft_consensus.cc:2781] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:00.718281 16306 raft_consensus.cc:686] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 LEADER]: Becoming Leader. State: Replica: 055a3aea67e34bafa5dbb2a0a347e50a, State: Running, Role: LEADER
I20260926 07:28:00.718657 16306 consensus_queue.cc:237] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } } peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } }
I20260926 07:28:00.721985 15823 catalog_manager.cc:5495] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a reported cstate change: term changed from 0 to 1, leader changed from <none> to 055a3aea67e34bafa5dbb2a0a347e50a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } health_report { overall_health: UNKNOWN } } }
W20260926 07:28:00.735158 16284 tablet.cc:2369] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:28:00.768029 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.763834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.0.98.1:46775)
0926 07:28:00.763982 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:00.763987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:00.764001 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:28:00.764181 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:00.764185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:00.764245 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:00.764369 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:28:00.764639 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.765166 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.766153 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:00.766438 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:00.766560 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:00.767117 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:28:00.767601 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:28:00.767752 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:00.767803 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 07:28:00.767880 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 07:28:00.768518 16150 tablet.cc:2369] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:28:00.773819 16237 raft_consensus.cc:1261] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3 [term 1 FOLLOWER]: Refusing update from remote peer 055a3aea67e34bafa5dbb2a0a347e50a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:00.773819 16103 raft_consensus.cc:1261] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f [term 1 FOLLOWER]: Refusing update from remote peer 055a3aea67e34bafa5dbb2a0a347e50a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 07:28:00.774590 16016 tablet.cc:2369] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:28:00.774530 16306 consensus_queue.cc:1035] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:00.774751 16311 consensus_queue.cc:1035] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:00.781651 16314 mvcc.cc:205] Tried to move back new op lower bound from 7333509860442472448 to 7333509860226899968. Current Snapshot: MvccSnapshot[applied={T|T < 7333509860442472448}]
I20260926 07:28:00.782471 16315 mvcc.cc:205] Tried to move back new op lower bound from 7333509860442472448 to 7333509860226899968. Current Snapshot: MvccSnapshot[applied={T|T < 7333509860442472448}]
I20260926 07:28:00.790364 15823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:00.790513 15823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:00.792958 15823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Sending DeleteTablet for 3 replicas of tablet f05ca8a0ae904c3e8752352b8a19cc6b
I20260926 07:28:00.793704 15949 tablet_service.cc:1514] Processing DeleteTablet for tablet f05ca8a0ae904c3e8752352b8a19cc6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53408
I20260926 07:28:00.793717 16217 tablet_service.cc:1514] Processing DeleteTablet for tablet f05ca8a0ae904c3e8752352b8a19cc6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52906
I20260926 07:28:00.794215 16324 tablet_replica.cc:330] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3: stopping tablet replica
I20260926 07:28:00.794466 16083 tablet_service.cc:1514] Processing DeleteTablet for tablet f05ca8a0ae904c3e8752352b8a19cc6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47664
I20260926 07:28:00.794512 16325 tablet_replica.cc:330] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: stopping tablet replica
I20260926 07:28:00.794591 16324 raft_consensus.cc:2229] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:00.794797 16325 raft_consensus.cc:2229] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:28:00.794898 16324 raft_consensus.cc:2258] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:00.794975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 15886
I20260926 07:28:00.796227 16326 tablet_replica.cc:330] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: stopping tablet replica
I20260926 07:28:00.796536 16326 raft_consensus.cc:2229] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:00.796859 16326 raft_consensus.cc:2258] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:00.797255 16324 ts_tablet_manager.cc:1905] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:00.798249 16326 ts_tablet_manager.cc:1905] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:00.800096 16324 ts_tablet_manager.cc:1918] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:00.800207 16324 log.cc:1196] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/f05ca8a0ae904c3e8752352b8a19cc6b
I20260926 07:28:00.800570 16324 ts_tablet_manager.cc:1939] T f05ca8a0ae904c3e8752352b8a19cc6b P 4901232023b24e9c91576f95abce51a3: Deleting consensus metadata
I20260926 07:28:00.801115 16326 ts_tablet_manager.cc:1918] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:00.801285 16326 log.cc:1196] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/f05ca8a0ae904c3e8752352b8a19cc6b
I20260926 07:28:00.801487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 16020
I20260926 07:28:00.802542 15808 catalog_manager.cc:4841] TS 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189): tablet f05ca8a0ae904c3e8752352b8a19cc6b (table test-table [id=650feac167a84063a72373a489569d13]) successfully deleted
W20260926 07:28:00.802609 15810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46775 (error 108)
I20260926 07:28:00.803714 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.803306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46775 (local address 127.0.0.1:53418)
0926 07:28:00.803484 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:00.803588 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 07:28:00.803897 15810 catalog_manager.cc:4584] TS 055a3aea67e34bafa5dbb2a0a347e50a (127.0.98.1:46775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f05ca8a0ae904c3e8752352b8a19cc6b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46775: connect: Connection refused (error 111)
I20260926 07:28:00.807298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 16154
I20260926 07:28:00.807837 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:00.807500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38273 (local address 127.0.0.1:47672)
0926 07:28:00.807687 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:00.807745 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 07:28:00.812898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 15792
I20260926 07:28:20.819595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46307
--webserver_interface=127.0.98.62
--webserver_port=36879
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:20.929661 16328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:20.929920 16328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:20.929961 16328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:20.933876 16328 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:28:20.933923 16328 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:28:20.933939 16328 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:28:20.933954 16328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:28:20.933970 16328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:20.933987 16328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:20.934003 16328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:28:20.934018 16328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:28:20.939132 16328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:20.940088 16328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:20.941031 16328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:20.954061 16334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:20.954071 16337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:20.954150 16328 server_base.cc:1034] running on GCE node
W20260926 07:28:20.954144 16335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:20.954968 16328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:20.956193 16328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:20.957396 16328 hybrid_clock.cc:648] HybridClock initialized: now 1790407700957356 us; error 53 us; skew 500 ppm
Sep 26 07:28:20 dist-test-slave-g04g krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:20.960397 16328 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:28:20.961350 16328 webserver.cc:460] Webserver started at http://127.0.98.62:36879/ using document root <none> and password file <none>
I20260926 07:28:20.961949 16328 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:20.962049 16328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:20.966351 16328 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:28:20.968665 16344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:20.969974 16328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:28:20.970139 16328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "c453ab1ec5a24671816da058aeee41ea"
format_stamp: "Formatted at 2026-09-26 07:27:59 on dist-test-slave-g04g"
server_key: "09a47ae2e79fa689f38f862af565e839"
server_key_iv: "999d47b10e7af5cba7286a7a1b795f07"
server_key_version: "encryptionkey@0"
I20260926 07:28:20.970775 16328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:20.995360 16328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:20.998801 16328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:20.999025 16328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:21.006788 16328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46307
I20260926 07:28:21.007135 16396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46307 every 8 connection(s)
I20260926 07:28:21.008044 16328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:28:21.012576 16397 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:28:21.014191 16397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Bootstrap starting.
I20260926 07:28:21.016117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 16328
I20260926 07:28:21.016543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46775
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46307
--webserver_port=37575
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:28:21.031169 16397 log.cc:824] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:21.039567 16397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:28:21.039968 16397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Bootstrap complete.
I20260926 07:28:21.043972 16397 raft_consensus.cc:348] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260926 07:28:21.044344 16397 raft_consensus.cc:729] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c453ab1ec5a24671816da058aeee41ea, State: Initialized, Role: FOLLOWER
I20260926 07:28:21.044827 16397 consensus_queue.cc:260] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260926 07:28:21.044938 16397 raft_consensus.cc:388] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:28:21.045027 16397 raft_consensus.cc:482] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:28:21.045125 16397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:28:21.046924 16397 raft_consensus.cc:504] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260926 07:28:21.047276 16397 leader_election.cc:302] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c453ab1ec5a24671816da058aeee41ea; no voters:
I20260926 07:28:21.047608 16397 leader_election.cc:288] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:28:21.047684 16401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:28:21.047899 16401 raft_consensus.cc:686] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 2 LEADER]: Becoming Leader. State: Replica: c453ab1ec5a24671816da058aeee41ea, State: Running, Role: LEADER
I20260926 07:28:21.048183 16401 consensus_queue.cc:237] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260926 07:28:21.048811 16397 sys_catalog.cc:564] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:28:21.049466 16402 sys_catalog.cc:455] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader c453ab1ec5a24671816da058aeee41ea. Latest consensus state: current_term: 2 leader_uuid: "c453ab1ec5a24671816da058aeee41ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } } }
I20260926 07:28:21.049587 16402 sys_catalog.cc:458] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:21.050182 16403 sys_catalog.cc:455] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c453ab1ec5a24671816da058aeee41ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } } }
I20260926 07:28:21.050418 16403 sys_catalog.cc:458] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:21.050500 16408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:28:21.055084 16408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:28:21.055688 16408 catalog_manager.cc:1259] Loaded cluster ID: 96c29ba8e487444aaadf6c3c322a6a95
I20260926 07:28:21.055747 16408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:28:21.058182 16408 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:28:21.058885 16408 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Loaded TSK: 0
I20260926 07:28:21.082405 16408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Generated new TSK 1
I20260926 07:28:21.082682 16408 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:28:21.133600 16399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:21.133901 16399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:21.133975 16399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:21.138079 16399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:21.138167 16399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:21.138252 16399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:28:21.143662 16399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37575
--tserver_master_addrs=127.0.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:21.144744 16399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:21.145766 16399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:21.156720 16424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:21.156692 16427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:21.156738 16425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:21.157152 16399 server_base.cc:1034] running on GCE node
I20260926 07:28:21.157541 16399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:21.158166 16399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:21.159363 16399 hybrid_clock.cc:648] HybridClock initialized: now 1790407701159332 us; error 49 us; skew 500 ppm
Sep 26 07:28:21 dist-test-slave-g04g krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:21.162391 16399 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:28:21.163486 16399 webserver.cc:460] Webserver started at http://127.0.98.1:37575/ using document root <none> and password file <none>
I20260926 07:28:21.164179 16399 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:21.164291 16399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:21.169299 16399 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:28:21.171954 16434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:21.173465 16399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:28:21.173584 16399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "055a3aea67e34bafa5dbb2a0a347e50a"
format_stamp: "Formatted at 2026-09-26 07:27:59 on dist-test-slave-g04g"
server_key: "cb0cadcecf90173e0630443a57255e0b"
server_key_iv: "6157525abbd170438f6db96de47ef4ea"
server_key_version: "encryptionkey@0"
I20260926 07:28:21.174285 16399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:21.190102 16399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:21.193552 16399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:21.193727 16399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:21.194334 16399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:21.195508 16441 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:28:21.199443 16399 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:28:21.199509 16399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 07:28:21.199576 16399 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:28:21.201627 16399 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:28:21.201682 16399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 07:28:21.201948 16441 tablet_bootstrap.cc:492] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Bootstrap starting.
I20260926 07:28:21.212093 16399 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46775
I20260926 07:28:21.212224 16548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46775 every 8 connection(s)
I20260926 07:28:21.213143 16399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:28:21.217103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 16399
I20260926 07:28:21.217576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38273
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46307
--webserver_port=45313
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:28:21.221793 16441 log.cc:824] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:28:21 dist-test-slave-g04g krb5kdc[15779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:28:21.229597 16441 tablet_bootstrap.cc:492] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:28:21.230038 16441 tablet_bootstrap.cc:492] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Bootstrap complete.
I20260926 07:28:21.231307 16441 ts_tablet_manager.cc:1397] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.011s
I20260926 07:28:21.233630 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.215368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49575 (local address 127.0.98.62:46307)
0926 07:28:21.216320 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.216337 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.216765 (+ 428us) server_negotiation.cc:407] Connection header received
0926 07:28:21.218221 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.218236 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.218608 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.218951 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:21.220564 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.222007 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.223596 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.224407 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.224609 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.227728 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:21.227734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:21.227749 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:21.227781 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:21.230010 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:21.230722 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:21.230731 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:21.230739 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:21.230902 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:21.231314 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:21.231320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:21.231325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:21.231760 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:21.231902 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:21.232525 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 07:28:21.232822 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":115,"threads_started":1}
I20260926 07:28:21.233999 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.215299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:49575)
0926 07:28:21.216582 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:21.216631 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:21.217680 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:21.219390 (+ 1710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:21.219398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:21.219823 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:21.220383 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.220400 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.222202 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.222205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.223382 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.223391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.224566 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.224570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.224753 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.225483 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:21.225502 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:21.227496 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:21.230177 (+ 2681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:21.230184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:21.230197 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:21.230582 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:21.231002 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:21.231005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:21.231007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:21.231200 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:21.231880 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:21.231886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:21.232175 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 07:28:21.232361 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 07:28:21.232689 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"thread_start_us":132,"threads_started":1}
I20260926 07:28:21.234117 16441 raft_consensus.cc:348] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } } peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } }
I20260926 07:28:21.234642 16441 raft_consensus.cc:729] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055a3aea67e34bafa5dbb2a0a347e50a, State: Initialized, Role: FOLLOWER
I20260926 07:28:21.235175 16441 consensus_queue.cc:260] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } } peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } }
I20260926 07:28:21.236225 16441 ts_tablet_manager.cc:1428] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:28:21.236493 16549 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46307
I20260926 07:28:21.236716 16549 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:21.237222 16549 heartbeater.cc:511] Master 127.0.98.62:46307 requested a full tablet report, sending...
I20260926 07:28:21.239692 16361 ts_manager.cc:194] Registered new tserver with Master: 055a3aea67e34bafa5dbb2a0a347e50a (127.0.98.1:46775)
I20260926 07:28:21.242890 16361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49575
I20260926 07:28:21.245478 16549 heartbeater.cc:503] Master 127.0.98.62:46307 was elected leader, sending a full tablet report...
Sep 26 07:28:21 dist-test-slave-g04g krb5kdc[15779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 07:28:21.252841 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.242359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46775 (local address 127.0.0.1:47842)
0926 07:28:21.242734 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:21.242780 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:21.242907 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:21.243578 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:21.243587 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:21.244019 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:21.244118 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.244124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.245055 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.245060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.245751 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.245761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.246476 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.246492 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.246619 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.247333 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:21.247344 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:21.248887 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:21.251099 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:21.251108 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:21.251113 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:21.251353 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:21.251690 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:21.251692 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:21.251693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:21.251773 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:21.252294 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:21.252299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:21.252417 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 07:28:21.252527 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:28:21.252708 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":111,"threads_started":1}
I20260926 07:28:21.252992 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.242455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.0.98.1:46775)
0926 07:28:21.243123 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.243134 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.243156 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:28:21.243221 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.243228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.243418 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.243565 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:21.244236 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.244942 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.245887 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.246368 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.246476 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.249029 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:21.249039 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:21.249051 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:21.249078 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:21.250939 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:21.251478 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:21.251486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:21.251490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:21.251591 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:21.251868 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:21.251871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:21.251873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:21.252178 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:21.252313 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:21.252654 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 07:28:21.252826 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":94,"threads_started":1}
I20260926 07:28:21.254201 16483 tablet_service.cc:1514] Processing DeleteTablet for tablet f05ca8a0ae904c3e8752352b8a19cc6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47842
I20260926 07:28:21.255223 16558 tablet_replica.cc:330] stopping tablet replica
I20260926 07:28:21.255510 16558 raft_consensus.cc:2229] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:21.255755 16558 raft_consensus.cc:2258] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:21.256677 16558 ts_tablet_manager.cc:1905] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:21.260344 16558 ts_tablet_manager.cc:1918] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:21.260454 16558 log.cc:1196] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal/wals/f05ca8a0ae904c3e8752352b8a19cc6b
I20260926 07:28:21.260718 16558 ts_tablet_manager.cc:1939] T f05ca8a0ae904c3e8752352b8a19cc6b P 055a3aea67e34bafa5dbb2a0a347e50a: Deleting consensus metadata
I20260926 07:28:21.262454 16347 catalog_manager.cc:4841] TS 055a3aea67e34bafa5dbb2a0a347e50a (127.0.98.1:46775): tablet f05ca8a0ae904c3e8752352b8a19cc6b (table test-table [id=650feac167a84063a72373a489569d13]) successfully deleted
W20260926 07:28:21.338814 16553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:21.339115 16553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:21.339164 16553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:21.343116 16553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:21.343168 16553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:21.343223 16553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:28:21.348309 16553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45313
--tserver_master_addrs=127.0.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:21.349337 16553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:21.350347 16553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:21.360301 16566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:21.360301 16563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:21.360432 16564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:21.360674 16553 server_base.cc:1034] running on GCE node
I20260926 07:28:21.361126 16553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:21.361740 16553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:21.362920 16553 hybrid_clock.cc:648] HybridClock initialized: now 1790407701362900 us; error 38 us; skew 500 ppm
Sep 26 07:28:21 dist-test-slave-g04g krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:21.365556 16553 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:28:21.366374 16553 webserver.cc:460] Webserver started at http://127.0.98.2:45313/ using document root <none> and password file <none>
I20260926 07:28:21.366911 16553 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:21.366983 16553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:21.370662 16553 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:28:21.372789 16573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:21.373981 16553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 07:28:21.374080 16553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "2ce29db40e9a47169f9ec63fccd90d1f"
format_stamp: "Formatted at 2026-09-26 07:28:00 on dist-test-slave-g04g"
server_key: "c94fe220c246746b3ad964b665618665"
server_key_iv: "f97fd1d65c0d1152fce970dff75bf1ec"
server_key_version: "encryptionkey@0"
I20260926 07:28:21.374676 16553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:21.393498 16553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:21.396140 16553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:21.396283 16553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:21.396778 16553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:21.397773 16580 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 07:28:21.401372 16580 ts_tablet_manager.cc:1852] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 07:28:21.401739 16580 ts_tablet_manager.cc:1905] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:21.405730 16580 ts_tablet_manager.cc:1918] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:21.405820 16580 log.cc:1196] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/f05ca8a0ae904c3e8752352b8a19cc6b
I20260926 07:28:21.406180 16580 ts_tablet_manager.cc:1939] T f05ca8a0ae904c3e8752352b8a19cc6b P 2ce29db40e9a47169f9ec63fccd90d1f: Deleting consensus metadata
I20260926 07:28:21.406478 16553 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 07:28:21.406553 16553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260926 07:28:21.406648 16553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:21.406687 16553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:21.415840 16553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38273
I20260926 07:28:21.415879 16687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38273 every 8 connection(s)
I20260926 07:28:21.416639 16553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:28:21.417402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 16553
I20260926 07:28:21.417817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33189
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46307
--webserver_port=36701
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:28:21 dist-test-slave-g04g krb5kdc[15779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:28:21.431985 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.418524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46585 (local address 127.0.98.62:46307)
0926 07:28:21.418676 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.418681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.419748 (+ 1067us) server_negotiation.cc:407] Connection header received
0926 07:28:21.420633 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.420636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.420684 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.420805 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:21.422380 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.422960 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.424828 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.425119 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.425221 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.428251 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:21.428255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:21.428261 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:21.428300 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:21.430032 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:21.430521 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:21.430524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:21.430526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:21.430608 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:21.430934 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:21.430937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:21.430939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:21.431166 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:21.431258 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:21.431707 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 07:28:21.431824 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:28:21.432696 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.418965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:46585)
0926 07:28:21.419578 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:21.419628 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:21.420430 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:21.420988 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:21.420999 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:21.421498 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:21.422184 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.422209 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.423123 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.423128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.424689 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.424703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.425285 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.425292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.425531 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.426234 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:21.426259 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:21.428083 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:21.430159 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:21.430164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:21.430174 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:21.430411 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:21.430715 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:21.430717 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:21.430718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:21.430842 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:21.431258 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:21.431262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:21.431482 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 07:28:21.431626 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 07:28:21.431847 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":139,"threads_started":1}
I20260926 07:28:21.433806 16688 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46307
I20260926 07:28:21.434036 16688 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:21.434484 16688 heartbeater.cc:511] Master 127.0.98.62:46307 requested a full tablet report, sending...
I20260926 07:28:21.435467 16361 ts_manager.cc:194] Registered new tserver with Master: 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273)
I20260926 07:28:21.436021 16361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46585
W20260926 07:28:21.530294 16690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:21.530587 16690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:21.530632 16690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:21.534675 16690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:21.534744 16690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:21.534847 16690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:28:21.540210 16690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36701
--tserver_master_addrs=127.0.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:21.541330 16690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:21.542410 16690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:21.553653 16696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:21.553640 16697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:21.553705 16690 server_base.cc:1034] running on GCE node
W20260926 07:28:21.553642 16699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:21.554401 16690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:21.555197 16690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:21.556414 16690 hybrid_clock.cc:648] HybridClock initialized: now 1790407701556397 us; error 33 us; skew 500 ppm
Sep 26 07:28:21 dist-test-slave-g04g krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:21.559509 16690 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:28:21.560482 16690 webserver.cc:460] Webserver started at http://127.0.98.3:36701/ using document root <none> and password file <none>
I20260926 07:28:21.561123 16690 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:21.561239 16690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:21.565470 16690 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:28:21.567703 16706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:21.569069 16690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:28:21.569209 16690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "4901232023b24e9c91576f95abce51a3"
format_stamp: "Formatted at 2026-09-26 07:28:00 on dist-test-slave-g04g"
server_key: "553fdaef3b4a6c6d48a94488424c7fb4"
server_key_iv: "10567e9a6a584f375670ddc71da3e412"
server_key_version: "encryptionkey@0"
I20260926 07:28:21.569803 16690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:21.590242 16690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:21.593735 16690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:21.593971 16690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:21.594621 16690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:21.595585 16690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:21.595660 16690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:21.595734 16690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:21.595796 16690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:21.605930 16690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33189
I20260926 07:28:21.605950 16819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33189 every 8 connection(s)
I20260926 07:28:21.607023 16690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:28:21.615307 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 16690
Sep 26 07:28:21 dist-test-slave-g04g krb5kdc[15779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:28:21.621243 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.608958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57115 (local address 127.0.98.62:46307)
0926 07:28:21.609105 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.609108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.610284 (+ 1176us) server_negotiation.cc:407] Connection header received
0926 07:28:21.611262 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.611266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.611325 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.611422 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:21.612618 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.613151 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.614291 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.614523 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.614635 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.617460 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:21.617464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:21.617467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:21.617490 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:21.619113 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:21.619699 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:21.619704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:21.619706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:21.619818 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:21.620168 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:21.620173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:21.620175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:21.620344 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:21.620454 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:21.620972 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 07:28:21.621087 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:28:21.622680 16822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.609448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:57115)
0926 07:28:21.610122 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:21.610175 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:21.611015 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:21.611611 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:21.611618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:21.611985 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:21.612459 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.612479 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.613282 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.613285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.614168 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.614177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.614673 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.614677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.614871 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.615553 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:21.615578 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:21.617299 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:21.619270 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:21.619279 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:21.619292 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:21.619555 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:21.619924 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:21.619926 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:21.619928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:21.620074 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:21.620453 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:21.620460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:21.620717 (+ 257us) client_negotiation.cc:769] Sending connection context
0926 07:28:21.620924 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 07:28:21.621315 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":120,"threads_started":1}
I20260926 07:28:21.624012 16820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46307
I20260926 07:28:21.624275 16820 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:21.624699 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.616040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.0.98.62:46307)
0926 07:28:21.616328 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.616334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.616494 (+ 160us) server_negotiation.cc:407] Connection header received
0926 07:28:21.616606 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.616609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.616665 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.616760 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:21.617560 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.618308 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.619317 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.619561 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.619672 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.620985 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:21.620989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:21.620992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:21.621020 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:21.622905 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:21.623431 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:21.623437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:21.623442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:21.623549 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:21.623823 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:21.623826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:21.623828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:21.624034 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:21.624244 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:21.624460 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 07:28:21.624556 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"spinlock_wait_cycles":1920,"thread_start_us":106,"threads_started":1}
I20260926 07:28:21.624917 16820 heartbeater.cc:511] Master 127.0.98.62:46307 requested a full tablet report, sending...
I20260926 07:28:21.625841 16359 ts_manager.cc:194] Registered new tserver with Master: 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189)
I20260926 07:28:21.626456 16359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57115
I20260926 07:28:21.638124 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:28:21.642125 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.638773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.0.98.62:46307)
0926 07:28:21.638912 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.638916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.639113 (+ 197us) server_negotiation.cc:407] Connection header received
0926 07:28:21.639272 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.639274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.639392 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.639485 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:28:21.639699 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.640222 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.641072 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.641300 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.641405 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.641647 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:28:21.641810 (+ 163us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 07:28:21.642026 (+ 216us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36842: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 07:28:21.642239 16823 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36842: token expired
W20260926 07:28:21.642591 16287 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:46307)
I20260926 07:28:21.650128 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.642972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36850 (local address 127.0.98.62:46307)
0926 07:28:21.643098 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.643101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.643146 (+ 45us) server_negotiation.cc:407] Connection header received
0926 07:28:21.643239 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.643241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.643276 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.643336 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:21.643908 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.644397 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.645249 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.645461 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.645538 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.646801 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:21.646804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:21.646806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:21.646828 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:21.648521 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:21.648958 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:21.648962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:21.648963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:21.649052 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:21.649355 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:21.649357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:21.649359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:21.649576 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:21.649683 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:21.649920 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 07:28:21.650012 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:28:21.652977 16359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:28:21.656570 16359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:28:21.664613 16483 tablet_service.cc:1467] Processing CreateTablet for tablet cc8d0eb903444e90a328e4b9dfc90a4e (DEFAULT_TABLE table=test-table [id=6e47cbfa9c4c40e6a33e7e15032f0f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:21.665330 16483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8d0eb903444e90a328e4b9dfc90a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:21.668385 16441 tablet_bootstrap.cc:492] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a: Bootstrap starting.
I20260926 07:28:21.669407 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.663427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38273 (local address 127.0.0.1:57498)
0926 07:28:21.663560 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:21.663577 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:21.663665 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:21.664598 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:21.664601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:21.664619 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:21.664734 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.664740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.666445 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.666449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.667754 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.667764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.668861 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.668864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.668977 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.669072 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 07:28:21.669153 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:28:21.669271 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":98}
I20260926 07:28:21.669447 16441 tablet_bootstrap.cc:654] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:21.669751 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.664252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.0.98.3:33189)
0926 07:28:21.664713 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.664722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.664743 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:28:21.664832 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.664838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.665063 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.665232 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:21.665488 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.666445 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.667996 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.668849 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.669278 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.669441 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:21.669501 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:28:21.669616 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":128,"threads_started":1}
I20260926 07:28:21.669921 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.663988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33189 (local address 127.0.0.1:47424)
0926 07:28:21.664336 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:21.664354 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:21.664455 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:21.665201 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:21.665204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:21.665223 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:21.665334 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.665341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.666569 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.666573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.667835 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.667849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.668974 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.668977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.669076 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.669094 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:28:21.669142 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 07:28:21.669784 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":552,"thread_start_us":116,"threads_started":1}
I20260926 07:28:21.670553 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.663682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.0.98.2:38273)
0926 07:28:21.664030 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.664043 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.664068 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:28:21.664148 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.664154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.664416 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.664900 (+ 484us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:21.665080 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.666306 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.667903 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.668685 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.668845 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.669014 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:21.670203 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0926 07:28:21.670332 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":133,"threads_started":1}
I20260926 07:28:21.671236 16441 tablet_bootstrap.cc:492] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a: No bootstrap required, opened a new log
I20260926 07:28:21.671361 16441 ts_tablet_manager.cc:1397] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260926 07:28:21.671864 16441 raft_consensus.cc:348] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:21.672005 16441 raft_consensus.cc:374] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:21.672049 16441 raft_consensus.cc:729] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055a3aea67e34bafa5dbb2a0a347e50a, State: Initialized, Role: FOLLOWER
I20260926 07:28:21.672183 16441 consensus_queue.cc:260] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:21.671880 16622 tablet_service.cc:1467] Processing CreateTablet for tablet cc8d0eb903444e90a328e4b9dfc90a4e (DEFAULT_TABLE table=test-table [id=6e47cbfa9c4c40e6a33e7e15032f0f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:21.672459 16441 ts_tablet_manager.cc:1428] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:28:21.673079 16622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8d0eb903444e90a328e4b9dfc90a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:21.673761 16754 tablet_service.cc:1467] Processing CreateTablet for tablet cc8d0eb903444e90a328e4b9dfc90a4e (DEFAULT_TABLE table=test-table [id=6e47cbfa9c4c40e6a33e7e15032f0f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:21.674854 16754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8d0eb903444e90a328e4b9dfc90a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:21.677563 16580 tablet_bootstrap.cc:492] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f: Bootstrap starting.
I20260926 07:28:21.679966 16580 tablet_bootstrap.cc:654] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:21.680922 16580 log.cc:824] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:21.683010 16834 tablet_bootstrap.cc:492] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3: Bootstrap starting.
I20260926 07:28:21.683151 16580 tablet_bootstrap.cc:492] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f: No bootstrap required, opened a new log
I20260926 07:28:21.683352 16580 ts_tablet_manager.cc:1397] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:28:21.685256 16834 tablet_bootstrap.cc:654] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:21.686295 16580 raft_consensus.cc:348] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:21.686473 16834 log.cc:824] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:21.686558 16580 raft_consensus.cc:374] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:21.686616 16580 raft_consensus.cc:729] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce29db40e9a47169f9ec63fccd90d1f, State: Initialized, Role: FOLLOWER
I20260926 07:28:21.687208 16580 consensus_queue.cc:260] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:21.687934 16688 heartbeater.cc:503] Master 127.0.98.62:46307 was elected leader, sending a full tablet report...
I20260926 07:28:21.688189 16580 ts_tablet_manager.cc:1428] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:28:21.688767 16834 tablet_bootstrap.cc:492] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3: No bootstrap required, opened a new log
I20260926 07:28:21.689009 16834 ts_tablet_manager.cc:1397] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 07:28:21.693305 16834 raft_consensus.cc:348] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:21.693610 16834 raft_consensus.cc:374] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:21.693694 16834 raft_consensus.cc:729] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4901232023b24e9c91576f95abce51a3, State: Initialized, Role: FOLLOWER
I20260926 07:28:21.694183 16834 consensus_queue.cc:260] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:21.694921 16820 heartbeater.cc:503] Master 127.0.98.62:46307 was elected leader, sending a full tablet report...
I20260926 07:28:21.695380 16834 ts_tablet_manager.cc:1428] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
W20260926 07:28:21.714740 16550 tablet.cc:2369] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:28:21.858072 16821 tablet.cc:2369] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:28:21.889698 16555 raft_consensus.cc:482] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:21.889989 16555 raft_consensus.cc:504] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:21.891438 16555 leader_election.cc:288] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273), 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189)
I20260926 07:28:21.897042 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.891886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47887 (local address 127.0.98.3:33189)
0926 07:28:21.892022 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.892026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.892233 (+ 207us) server_negotiation.cc:407] Connection header received
0926 07:28:21.892443 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.892445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.892492 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.892597 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:21.893029 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.893808 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.895661 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.896283 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.896406 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.896476 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:21.896829 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 07:28:21.896912 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:28:21.897042 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.891912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48315 (local address 127.0.98.2:38273)
0926 07:28:21.892038 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.892044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.892308 (+ 264us) server_negotiation.cc:407] Connection header received
0926 07:28:21.892432 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.892435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.892492 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.892597 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:21.893029 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.893808 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.895661 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.896309 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.896406 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.896476 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:21.896809 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 07:28:21.896894 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:28:21.897166 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.891735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33189 (local address 127.0.98.1:47887)
0926 07:28:21.892131 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:21.892147 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:21.892285 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:21.892627 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:21.892630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:21.892703 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:21.892903 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.892912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.893944 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.893947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.895509 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.895521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.896451 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.896455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.896645 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.896693 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 07:28:21.896785 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:28:21.896914 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":282,"spinlock_wait_cycles":19712,"thread_start_us":94,"threads_started":1}
I20260926 07:28:21.897184 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.891735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38273 (local address 127.0.98.1:48315)
0926 07:28:21.892230 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:21.892242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:21.892314 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:21.892627 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:21.892630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:21.892653 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:21.892903 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.892912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.893944 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.893947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.895509 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:21.895521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.896451 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.896455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:21.896610 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.896693 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 07:28:21.896794 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 07:28:21.896926 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":334,"spinlock_wait_cycles":43904,"thread_start_us":76,"threads_started":1}
I20260926 07:28:21.898005 16774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8d0eb903444e90a328e4b9dfc90a4e" candidate_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4901232023b24e9c91576f95abce51a3" is_pre_election: true
I20260926 07:28:21.898021 16642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8d0eb903444e90a328e4b9dfc90a4e" candidate_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" is_pre_election: true
I20260926 07:28:21.898319 16774 raft_consensus.cc:2454] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 055a3aea67e34bafa5dbb2a0a347e50a in term 0.
I20260926 07:28:21.898319 16642 raft_consensus.cc:2454] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 055a3aea67e34bafa5dbb2a0a347e50a in term 0.
I20260926 07:28:21.898790 16438 leader_election.cc:302] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 055a3aea67e34bafa5dbb2a0a347e50a, 2ce29db40e9a47169f9ec63fccd90d1f, 4901232023b24e9c91576f95abce51a3; no voters:
I20260926 07:28:21.899127 16555 raft_consensus.cc:2781] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:28:21.899227 16555 raft_consensus.cc:482] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:28:21.899286 16555 raft_consensus.cc:3037] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:21.900310 16555 raft_consensus.cc:504] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:21.900674 16555 leader_election.cc:288] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 1 election: Requested vote from peers 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273), 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189)
I20260926 07:28:21.901044 16642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8d0eb903444e90a328e4b9dfc90a4e" candidate_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ce29db40e9a47169f9ec63fccd90d1f"
I20260926 07:28:21.901181 16642 raft_consensus.cc:3037] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:21.901180 16774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8d0eb903444e90a328e4b9dfc90a4e" candidate_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4901232023b24e9c91576f95abce51a3"
I20260926 07:28:21.901319 16774 raft_consensus.cc:3037] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:21.902316 16642 raft_consensus.cc:2454] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 055a3aea67e34bafa5dbb2a0a347e50a in term 1.
I20260926 07:28:21.902351 16774 raft_consensus.cc:2454] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 055a3aea67e34bafa5dbb2a0a347e50a in term 1.
I20260926 07:28:21.902675 16438 leader_election.cc:302] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055a3aea67e34bafa5dbb2a0a347e50a, 2ce29db40e9a47169f9ec63fccd90d1f; no voters:
I20260926 07:28:21.902845 16555 raft_consensus.cc:2781] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:21.903113 16555 raft_consensus.cc:686] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 LEADER]: Becoming Leader. State: Replica: 055a3aea67e34bafa5dbb2a0a347e50a, State: Running, Role: LEADER
I20260926 07:28:21.903641 16555 consensus_queue.cc:237] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:21.906755 16359 catalog_manager.cc:5495] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a reported cstate change: term changed from 0 to 1, leader changed from <none> to 055a3aea67e34bafa5dbb2a0a347e50a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } health_report { overall_health: HEALTHY } } }
W20260926 07:28:21.917753 16689 tablet.cc:2369] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:28:21.969050 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:21.965204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.0.98.1:46775)
0926 07:28:21.965549 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:21.965558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:21.965575 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:28:21.965671 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:21.965673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:21.965729 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:21.965843 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:28:21.966050 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.966581 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.967522 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:21.967747 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:21.967839 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:21.968087 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:28:21.968552 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:28:21.968642 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:21.968793 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 07:28:21.968846 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":131,"threads_started":1}
I20260926 07:28:21.973402 16774 raft_consensus.cc:1261] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3 [term 1 FOLLOWER]: Refusing update from remote peer 055a3aea67e34bafa5dbb2a0a347e50a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:21.973400 16642 raft_consensus.cc:1261] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f [term 1 FOLLOWER]: Refusing update from remote peer 055a3aea67e34bafa5dbb2a0a347e50a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:28:21.974149 16840 consensus_queue.cc:1035] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:21.974351 16555 consensus_queue.cc:1035] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:28:21.981755 16843 mvcc.cc:205] Tried to move back new op lower bound from 7333509947275874304 to 7333509947000356864. Current Snapshot: MvccSnapshot[applied={T|T < 7333509947275874304}]
I20260926 07:28:21.984721 16845 mvcc.cc:205] Tried to move back new op lower bound from 7333509947275874304 to 7333509947000356864. Current Snapshot: MvccSnapshot[applied={T|T < 7333509947275874304 or (T in {7333509947275874304})}]
I20260926 07:28:21.991209 16359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:21.991412 16359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:28:21.993801 16359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Sending DeleteTablet for 3 replicas of tablet cc8d0eb903444e90a328e4b9dfc90a4e
I20260926 07:28:21.994620 16754 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8d0eb903444e90a328e4b9dfc90a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47424
I20260926 07:28:21.994860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 16399
I20260926 07:28:21.994872 16483 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8d0eb903444e90a328e4b9dfc90a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47842
I20260926 07:28:21.995162 16622 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8d0eb903444e90a328e4b9dfc90a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:28:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57498
I20260926 07:28:21.995846 16854 tablet_replica.cc:330] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f: stopping tablet replica
I20260926 07:28:21.996203 16854 raft_consensus.cc:2229] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:21.996390 16853 tablet_replica.cc:330] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3: stopping tablet replica
I20260926 07:28:21.996551 16854 raft_consensus.cc:2258] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:21.996680 16853 raft_consensus.cc:2229] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:28:21.997000 16853 raft_consensus.cc:2258] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:28:21.998143 16854 ts_tablet_manager.cc:1905] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:21.998457 16853 ts_tablet_manager.cc:1905] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:28:22.000389 16854 ts_tablet_manager.cc:1918] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:22.000479 16854 log.cc:1196] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/cc8d0eb903444e90a328e4b9dfc90a4e
I20260926 07:28:22.000715 16854 ts_tablet_manager.cc:1939] T cc8d0eb903444e90a328e4b9dfc90a4e P 2ce29db40e9a47169f9ec63fccd90d1f: Deleting consensus metadata
I20260926 07:28:22.001885 16853 ts_tablet_manager.cc:1918] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:28:22.001991 16853 log.cc:1196] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/cc8d0eb903444e90a328e4b9dfc90a4e
I20260926 07:28:22.002182 16348 catalog_manager.cc:4841] TS 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273): tablet cc8d0eb903444e90a328e4b9dfc90a4e (table test-table [id=6e47cbfa9c4c40e6a33e7e15032f0f81]) successfully deleted
I20260926 07:28:22.002352 16853 ts_tablet_manager.cc:1939] T cc8d0eb903444e90a328e4b9dfc90a4e P 4901232023b24e9c91576f95abce51a3: Deleting consensus metadata
W20260926 07:28:22.003265 16347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46775 (error 108)
I20260926 07:28:22.003736 16345 catalog_manager.cc:4841] TS 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189): tablet cc8d0eb903444e90a328e4b9dfc90a4e (table test-table [id=6e47cbfa9c4c40e6a33e7e15032f0f81]) successfully deleted
I20260926 07:28:22.004326 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:22.004061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46775 (local address 127.0.0.1:47860)
0926 07:28:22.004170 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:22.004229 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 07:28:22.004429 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 16553
W20260926 07:28:22.004525 16347 catalog_manager.cc:4584] TS 055a3aea67e34bafa5dbb2a0a347e50a (127.0.98.1:46775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc8d0eb903444e90a328e4b9dfc90a4e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46775: connect: Connection refused (error 111)
I20260926 07:28:22.010860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 16690
I20260926 07:28:22.018842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 16328
I20260926 07:28:42.025584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46307
--webserver_interface=127.0.98.62
--webserver_port=36879
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:42.141741 16856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:42.142009 16856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:42.142051 16856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:42.146206 16856 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:28:42.146270 16856 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:28:42.146286 16856 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:28:42.146299 16856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:28:42.146317 16856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:42.146335 16856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:42.146350 16856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:28:42.146365 16856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:28:42.151609 16856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:42.152606 16856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:42.153604 16856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:42.162700 16862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:42.162763 16861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:42.162853 16864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:42.162956 16856 server_base.cc:1034] running on GCE node
I20260926 07:28:42.163601 16856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:42.164659 16856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:42.165817 16856 hybrid_clock.cc:648] HybridClock initialized: now 1790407722165805 us; error 28 us; skew 500 ppm
Sep 26 07:28:42 dist-test-slave-g04g krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:42.168608 16856 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:28:42.169423 16856 webserver.cc:460] Webserver started at http://127.0.98.62:36879/ using document root <none> and password file <none>
I20260926 07:28:42.169919 16856 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:42.169975 16856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:42.173699 16856 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:42.175818 16871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:42.177058 16856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:42.177242 16856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "c453ab1ec5a24671816da058aeee41ea"
format_stamp: "Formatted at 2026-09-26 07:27:59 on dist-test-slave-g04g"
server_key: "09a47ae2e79fa689f38f862af565e839"
server_key_iv: "999d47b10e7af5cba7286a7a1b795f07"
server_key_version: "encryptionkey@0"
I20260926 07:28:42.177783 16856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:42.190141 16856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:42.193780 16856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:42.193935 16856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:42.201440 16856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46307
I20260926 07:28:42.201725 16923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46307 every 8 connection(s)
I20260926 07:28:42.202663 16856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:28:42.207091 16924 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:28:42.208655 16924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Bootstrap starting.
I20260926 07:28:42.212586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 16856
I20260926 07:28:42.213059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46775
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46307
--webserver_port=37575
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:28:42.226974 16924 log.cc:824] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:42.238237 16924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:28:42.238602 16924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Bootstrap complete.
I20260926 07:28:42.242552 16924 raft_consensus.cc:348] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260926 07:28:42.242966 16924 raft_consensus.cc:729] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c453ab1ec5a24671816da058aeee41ea, State: Initialized, Role: FOLLOWER
I20260926 07:28:42.243503 16924 consensus_queue.cc:260] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260926 07:28:42.243659 16924 raft_consensus.cc:388] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:28:42.243744 16924 raft_consensus.cc:482] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:28:42.243858 16924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:28:42.245648 16924 raft_consensus.cc:504] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260926 07:28:42.245973 16924 leader_election.cc:302] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c453ab1ec5a24671816da058aeee41ea; no voters:
I20260926 07:28:42.246308 16924 leader_election.cc:288] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 07:28:42.246402 16928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:28:42.246618 16928 raft_consensus.cc:686] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [term 3 LEADER]: Becoming Leader. State: Replica: c453ab1ec5a24671816da058aeee41ea, State: Running, Role: LEADER
I20260926 07:28:42.246898 16928 consensus_queue.cc:237] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260926 07:28:42.247543 16924 sys_catalog.cc:564] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:28:42.247908 16928 sys_catalog.cc:455] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader c453ab1ec5a24671816da058aeee41ea. Latest consensus state: current_term: 3 leader_uuid: "c453ab1ec5a24671816da058aeee41ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } } }
I20260926 07:28:42.248016 16928 sys_catalog.cc:458] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:42.248450 16929 sys_catalog.cc:455] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c453ab1ec5a24671816da058aeee41ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c453ab1ec5a24671816da058aeee41ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } } }
I20260926 07:28:42.248499 16932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:28:42.249140 16929 sys_catalog.cc:458] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:42.253728 16932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:28:42.254400 16932 catalog_manager.cc:1259] Loaded cluster ID: 96c29ba8e487444aaadf6c3c322a6a95
I20260926 07:28:42.254480 16932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:28:42.256244 16932 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:28:42.256841 16932 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Loaded TSK: 0
I20260926 07:28:42.256932 16932 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c453ab1ec5a24671816da058aeee41ea: Loaded TSK: 1
F20260926 07:28:42.257596 16932 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0be7ff79ad google::LogMessage::Fail() at ??:0
@ 0x7f0be7ffa88a google::LogMessage::SendToLog() at ??:0
@ 0x7f0be7ff74de google::LogMessage::Flush() at ??:0
@ 0x7f0be7ff8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0beacbb18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0beacb3f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f0bf295c7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f0bf293699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0bf2937178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f0bf2978df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f0bf299782b std::function<>::operator()() at ??:0
@ 0x7f0bf2936b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f0bf2937f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0bf2934579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f0bf2978640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0bea2493d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0bea249c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f0bea24b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0bea23a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0beb87e6db start_thread at ??:0
@ 0x7f0be6af561f clone at ??:0
W20260926 07:28:42.365986 16926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:42.366338 16926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:42.366405 16926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:42.372555 16926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:42.372627 16926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:42.372723 16926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:28:42.380412 16926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37575
--tserver_master_addrs=127.0.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:42.381837 16926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:42.383200 16926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:42.393103 16971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:42.393103 16972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:42.393141 16974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:42.393481 16926 server_base.cc:1034] running on GCE node
I20260926 07:28:42.393896 16926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:42.394550 16926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:42.395728 16926 hybrid_clock.cc:648] HybridClock initialized: now 1790407722395720 us; error 41 us; skew 500 ppm
Sep 26 07:28:42 dist-test-slave-g04g krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:42.398351 16926 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:28:42.399372 16926 webserver.cc:460] Webserver started at http://127.0.98.1:37575/ using document root <none> and password file <none>
I20260926 07:28:42.399977 16926 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:42.400066 16926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:42.404080 16926 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:28:42.406167 16981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:42.407477 16926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:28:42.407588 16926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "055a3aea67e34bafa5dbb2a0a347e50a"
format_stamp: "Formatted at 2026-09-26 07:27:59 on dist-test-slave-g04g"
server_key: "cb0cadcecf90173e0630443a57255e0b"
server_key_iv: "6157525abbd170438f6db96de47ef4ea"
server_key_version: "encryptionkey@0"
I20260926 07:28:42.408102 16926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:42.419661 16926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:42.422823 16926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:42.423059 16926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:42.423731 16926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:42.424808 16988 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:28:42.428891 16926 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:28:42.428954 16926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260926 07:28:42.429000 16926 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:28:42.430624 16926 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:28:42.430677 16926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 07:28:42.430939 16988 tablet_bootstrap.cc:492] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a: Bootstrap starting.
I20260926 07:28:42.441321 16926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46775
I20260926 07:28:42.441517 17095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46775 every 8 connection(s)
I20260926 07:28:42.442556 16926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:28:42.450196 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 16926
I20260926 07:28:42.450699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38273
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46307
--webserver_port=45313
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:28:42.450976 16988 log.cc:824] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:42.459733 16988 tablet_bootstrap.cc:492] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:28:42.460170 16988 tablet_bootstrap.cc:492] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a: Bootstrap complete.
I20260926 07:28:42.461681 16988 ts_tablet_manager.cc:1397] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.004s
I20260926 07:28:42.465459 16988 raft_consensus.cc:348] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:42.465982 16988 raft_consensus.cc:729] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055a3aea67e34bafa5dbb2a0a347e50a, State: Initialized, Role: FOLLOWER
I20260926 07:28:42.466501 16988 consensus_queue.cc:260] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:42.467522 16988 ts_tablet_manager.cc:1428] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260926 07:28:42.491590 17098 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:28:42.444856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:57195)
0926 07:28:42.445448 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:42.445512 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:42.446550 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:42.490772 (+ 44222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":144,"threads_started":1}
W20260926 07:28:42.492072 17096 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46307 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46307: Client connection negotiation failed: client connection to 127.0.98.62:46307: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:28:42.492767 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:42.492517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:42233)
0926 07:28:42.492646 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:42.492717 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 07:28:42.493364 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:42.493169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:48737)
0926 07:28:42.493266 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:42.493310 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 07:28:42.493521 17096 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:28:42.567186 17100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:42.567451 17100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:42.567526 17100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:42.571700 17100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:42.571781 17100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:42.571875 17100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:28:42.577028 17100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45313
--tserver_master_addrs=127.0.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:42.578099 17100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:42.579191 17100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:42.588716 17109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:42.588922 17100 server_base.cc:1034] running on GCE node
W20260926 07:28:42.588852 17106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:42.589232 17107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:42.589653 17100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:42.590269 17100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:42.591445 17100 hybrid_clock.cc:648] HybridClock initialized: now 1790407722591406 us; error 55 us; skew 500 ppm
Sep 26 07:28:42 dist-test-slave-g04g krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:42.594080 17100 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:28:42.594933 17100 webserver.cc:460] Webserver started at http://127.0.98.2:45313/ using document root <none> and password file <none>
I20260926 07:28:42.595511 17100 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:42.595580 17100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:42.599335 17100 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 07:28:42.601441 17116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:42.602746 17100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:42.602869 17100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "2ce29db40e9a47169f9ec63fccd90d1f"
format_stamp: "Formatted at 2026-09-26 07:28:00 on dist-test-slave-g04g"
server_key: "c94fe220c246746b3ad964b665618665"
server_key_iv: "f97fd1d65c0d1152fce970dff75bf1ec"
server_key_version: "encryptionkey@0"
I20260926 07:28:42.603503 17100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:42.621408 17100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:42.625555 17100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:42.625775 17100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:42.626523 17100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:42.627547 17100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:42.627626 17100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:42.627691 17100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:42.627725 17100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:42.636969 17100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38273
I20260926 07:28:42.636993 17229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38273 every 8 connection(s)
I20260926 07:28:42.638000 17100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:28:42.639595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 17100
I20260926 07:28:42.640044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33189
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46307
--webserver_port=36701
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:28:42.641970 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:42.640303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:47699)
0926 07:28:42.640808 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:42.641309 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":127,"threads_started":1}
W20260926 07:28:42.642474 17230 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46307 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46307: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
I20260926 07:28:42.643112 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:42.642845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:58659)
0926 07:28:42.642954 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:42.643005 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 07:28:42.643859 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:42.643599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:41417)
0926 07:28:42.643717 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:42.643779 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 07:28:42.644059 17230 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:28:42.751216 17233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:42.751479 17233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:42.751554 17233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:42.755551 17233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:42.755626 17233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:42.755728 17233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:28:42.760720 17233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36701
--tserver_master_addrs=127.0.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:42.761684 17233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:42.762681 17233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:42.772095 17239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:42.772076 17241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:42.772110 17238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:42.772524 17233 server_base.cc:1034] running on GCE node
I20260926 07:28:42.772977 17233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:42.773607 17233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:42.774791 17233 hybrid_clock.cc:648] HybridClock initialized: now 1790407722774758 us; error 44 us; skew 500 ppm
Sep 26 07:28:42 dist-test-slave-g04g krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:42.777691 17233 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:28:42.778569 17233 webserver.cc:460] Webserver started at http://127.0.98.3:36701/ using document root <none> and password file <none>
I20260926 07:28:42.779157 17233 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:42.779258 17233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:42.783205 17233 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:28:42.785332 17248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:42.786607 17233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:42.786700 17233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "4901232023b24e9c91576f95abce51a3"
format_stamp: "Formatted at 2026-09-26 07:28:00 on dist-test-slave-g04g"
server_key: "553fdaef3b4a6c6d48a94488424c7fb4"
server_key_iv: "10567e9a6a584f375670ddc71da3e412"
server_key_version: "encryptionkey@0"
I20260926 07:28:42.787271 17233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:42.810297 17233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:42.813632 17233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:42.813848 17233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:42.814461 17233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:42.815335 17233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:42.815407 17233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:42.815476 17233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:42.815527 17233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:42.824368 17233 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33189
I20260926 07:28:42.824445 17361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33189 every 8 connection(s)
I20260926 07:28:42.825402 17233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:28:42.827212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 17233
I20260926 07:28:42.829592 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:42.827742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:49723)
0926 07:28:42.828763 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:42.828871 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":686,"threads_started":1}
W20260926 07:28:42.830040 17362 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46307 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46307: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
I20260926 07:28:42.830688 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:42.830449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:37677)
0926 07:28:42.830592 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:42.830637 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 07:28:42.831357 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:42.831176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:51219)
0926 07:28:42.831271 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:42.831312 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 07:28:42.831691 17362 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:28:43.494582 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:43.494130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:56549)
0926 07:28:43.494405 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:43.494465 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":72,"threads_started":1}
I20260926 07:28:43.644953 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:43.644598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:44397)
0926 07:28:43.644810 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:43.644861 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
I20260926 07:28:43.749434 17368 raft_consensus.cc:482] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:43.749624 17368 raft_consensus.cc:504] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:43.750698 17368 leader_election.cc:288] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273), 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189)
Sep 26 07:28:43 dist-test-slave-g04g krb5kdc[15779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 07:28:43 dist-test-slave-g04g krb5kdc[15779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 07:28:43.767962 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:43.751093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41433 (local address 127.0.98.2:38273)
0926 07:28:43.751498 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:43.751524 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:43.751570 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:28:43.752467 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:43.752476 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:43.752923 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:43.753248 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:43.755122 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:43.756727 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:43.758767 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:43.759487 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:43.759651 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:43.763348 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:43.763352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:43.763362 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:43.763389 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:43.765325 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:43.765972 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:43.765978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:43.765985 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:43.766100 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:43.766441 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:43.766446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:43.766448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:43.766839 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:43.766988 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:43.767544 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 07:28:43.767791 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":96,"threads_started":1}
I20260926 07:28:43.767951 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:43.750887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38273 (local address 127.0.98.1:41433)
0926 07:28:43.751129 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:43.751144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:43.751248 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:43.753350 (+ 2102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:43.753356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:43.754024 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:43.754911 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:43.754931 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:43.756885 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:43.756888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:43.758636 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:43.758647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:43.759648 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:43.759651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:43.760007 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:43.760800 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:43.760814 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:43.763166 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:43.765455 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:43.765459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:43.765470 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:43.765856 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:43.766200 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:43.766205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:43.766207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:43.766343 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:43.766960 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:43.766967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:43.767228 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 07:28:43.767417 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 07:28:43.767737 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":454,"spinlock_wait_cycles":20096}
I20260926 07:28:43.768055 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:43.751140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40633 (local address 127.0.98.3:33189)
0926 07:28:43.751534 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:43.751544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:43.751574 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:28:43.752470 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:43.752476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:43.752923 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:43.753248 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:43.755122 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:43.756727 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:43.758767 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:43.759495 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:43.759650 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:43.763113 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:43.763121 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:43.763137 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:43.763173 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:43.765294 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:43.765980 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:43.765986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:43.765992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:43.766113 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:43.766464 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:43.766468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:43.766470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:43.766846 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:43.766987 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:43.767594 (+ 607us) server_negotiation.cc:299] Negotiation successful
0926 07:28:43.767860 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":59,"threads_started":1}
I20260926 07:28:43.768703 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:43.750903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33189 (local address 127.0.98.1:40633)
0926 07:28:43.751258 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:43.751271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:43.751376 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:43.753350 (+ 1974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:43.753356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:43.754024 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:43.754911 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:43.754931 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:43.756885 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:43.756888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:43.758636 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:43.758647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:43.759635 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:43.759640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:43.759981 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:43.760772 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:43.760802 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:43.762791 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:43.765448 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:43.765455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:43.765470 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:43.765851 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:43.766229 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:43.766232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:43.766234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:43.766349 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:43.767057 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:43.767060 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:43.767231 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 07:28:43.767426 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 07:28:43.767838 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":580,"spinlock_wait_cycles":32256,"thread_start_us":71,"threads_started":1}
W20260926 07:28:43.770663 16985 leader_election.cc:341] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273): Not found: Tablet not found: cc8d0eb903444e90a328e4b9dfc90a4e
W20260926 07:28:43.770831 16982 leader_election.cc:341] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189): Not found: Tablet not found: cc8d0eb903444e90a328e4b9dfc90a4e
I20260926 07:28:43.770946 16982 leader_election.cc:302] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 055a3aea67e34bafa5dbb2a0a347e50a; no voters: 2ce29db40e9a47169f9ec63fccd90d1f, 4901232023b24e9c91576f95abce51a3
I20260926 07:28:43.771245 17368 raft_consensus.cc:2726] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:28:43.832664 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:43.832242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:60739)
0926 07:28:43.832502 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:43.832557 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":96,"threads_started":1}
I20260926 07:28:44.496132 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:44.495479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:53429)
0926 07:28:44.495749 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:44.495814 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260926 07:28:44.646049 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:44.645656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:34539)
0926 07:28:44.645886 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:44.645936 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260926 07:28:44.833707 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:44.833324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:51017)
0926 07:28:44.833551 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:44.833605 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":70,"threads_started":1}
I20260926 07:28:45.497368 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:45.496986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:55499)
0926 07:28:45.497217 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:45.497273 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260926 07:28:45.647166 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:45.646713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:49307)
0926 07:28:45.646950 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:45.647007 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260926 07:28:45.834864 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:45.834426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:49729)
0926 07:28:45.834702 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:45.834763 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260926 07:28:45.873304 17379 raft_consensus.cc:482] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:45.873584 17379 raft_consensus.cc:504] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:45.874169 17379 leader_election.cc:288] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273), 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189)
W20260926 07:28:45.875147 16985 leader_election.cc:341] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273): Not found: Tablet not found: cc8d0eb903444e90a328e4b9dfc90a4e
W20260926 07:28:45.875916 16982 leader_election.cc:341] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189): Not found: Tablet not found: cc8d0eb903444e90a328e4b9dfc90a4e
I20260926 07:28:45.876012 16982 leader_election.cc:302] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 055a3aea67e34bafa5dbb2a0a347e50a; no voters: 2ce29db40e9a47169f9ec63fccd90d1f, 4901232023b24e9c91576f95abce51a3
I20260926 07:28:45.876237 17379 raft_consensus.cc:2726] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:28:46.498579 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:46.498170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:45513)
0926 07:28:46.498423 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:46.498482 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
I20260926 07:28:46.648253 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:46.647867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:54581)
0926 07:28:46.648110 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:46.648164 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260926 07:28:46.836133 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:46.835720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:33507)
0926 07:28:46.835978 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:46.836045 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260926 07:28:47.500202 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:47.499635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:42743)
0926 07:28:47.500040 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:47.500102 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":90,"threads_started":1}
I20260926 07:28:47.649405 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:47.649019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:41777)
0926 07:28:47.649264 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:47.649317 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260926 07:28:47.837327 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:47.836931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:38771)
0926 07:28:47.837158 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:47.837232 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260926 07:28:48.501339 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:48.500964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:50959)
0926 07:28:48.501193 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:48.501245 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260926 07:28:48.650520 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:48.650140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:54083)
0926 07:28:48.650382 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:48.650435 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260926 07:28:48.716804 17388 raft_consensus.cc:482] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:48.716974 17388 raft_consensus.cc:504] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:48.717504 17388 leader_election.cc:288] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273), 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189)
W20260926 07:28:48.718366 16982 leader_election.cc:341] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189): Not found: Tablet not found: cc8d0eb903444e90a328e4b9dfc90a4e
W20260926 07:28:48.718475 16985 leader_election.cc:341] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273): Not found: Tablet not found: cc8d0eb903444e90a328e4b9dfc90a4e
I20260926 07:28:48.718694 16985 leader_election.cc:302] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 055a3aea67e34bafa5dbb2a0a347e50a; no voters: 2ce29db40e9a47169f9ec63fccd90d1f, 4901232023b24e9c91576f95abce51a3
I20260926 07:28:48.718868 17388 raft_consensus.cc:2726] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:28:48.838559 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:48.838075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:33017)
0926 07:28:48.838356 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:48.838437 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
I20260926 07:28:49.502667 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:49.502217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:40639)
0926 07:28:49.502506 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:49.502562 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":116,"threads_started":1}
I20260926 07:28:49.651732 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:49.651279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:50045)
0926 07:28:49.651571 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:49.651639 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
I20260926 07:28:49.839752 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:49.839327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:49995)
0926 07:28:49.839604 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:49.839660 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":67,"threads_started":1}
I20260926 07:28:50.503940 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:50.503373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:56967)
0926 07:28:50.503758 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:50.503818 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":76,"threads_started":1}
I20260926 07:28:50.652917 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:50.652489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:52615)
0926 07:28:50.652761 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:50.652822 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
I20260926 07:28:50.840844 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:50.840432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:59717)
0926 07:28:50.840703 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:50.840754 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
I20260926 07:28:51.505110 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:51.504712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:56609)
0926 07:28:51.504952 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:51.505008 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260926 07:28:51.654067 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:51.653675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:36515)
0926 07:28:51.653924 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:51.653977 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260926 07:28:51.842090 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:51.841646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:41953)
0926 07:28:51.841944 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:51.841999 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
I20260926 07:28:52.506680 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:52.506140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:33877)
0926 07:28:52.506481 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:52.506529 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":105,"threads_started":1}
I20260926 07:28:52.655419 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:52.654819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:41509)
0926 07:28:52.655262 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:52.655320 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":98,"threads_started":1}
I20260926 07:28:52.843304 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:52.842834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:36737)
0926 07:28:52.843151 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:52.843207 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260926 07:28:53.507887 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:53.507436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:57079)
0926 07:28:53.507670 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:53.507725 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260926 07:28:53.656565 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:53.656187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:39049)
0926 07:28:53.656428 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:53.656479 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":67,"threads_started":1}
I20260926 07:28:53.659389 17404 raft_consensus.cc:482] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:53.659513 17404 raft_consensus.cc:504] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:53.659933 17404 leader_election.cc:288] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273), 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189)
W20260926 07:28:53.660636 16985 leader_election.cc:341] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273): Not found: Tablet not found: cc8d0eb903444e90a328e4b9dfc90a4e
W20260926 07:28:53.660799 16982 leader_election.cc:341] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189): Not found: Tablet not found: cc8d0eb903444e90a328e4b9dfc90a4e
I20260926 07:28:53.660887 16982 leader_election.cc:302] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 055a3aea67e34bafa5dbb2a0a347e50a; no voters: 2ce29db40e9a47169f9ec63fccd90d1f, 4901232023b24e9c91576f95abce51a3
I20260926 07:28:53.661024 17404 raft_consensus.cc:2726] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:28:53.844405 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:53.844025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:59783)
0926 07:28:53.844256 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:53.844313 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
I20260926 07:28:54.509020 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:54.508633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:52359)
0926 07:28:54.508876 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:54.508929 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260926 07:28:54.657800 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:54.657310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:36869)
0926 07:28:54.657569 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:54.657653 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":99,"threads_started":1}
I20260926 07:28:54.845479 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:54.845088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:52731)
0926 07:28:54.845341 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:54.845391 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
I20260926 07:28:55.510097 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:55.509713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:41667)
0926 07:28:55.509947 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:55.510000 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260926 07:28:55.658991 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:55.658601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:41589)
0926 07:28:55.658852 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:55.658900 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
I20260926 07:28:55.846549 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:55.846171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:54685)
0926 07:28:55.846405 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:55.846456 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
I20260926 07:28:56.511277 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:56.510806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:42571)
0926 07:28:56.511130 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:56.511182 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260926 07:28:56.660228 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:56.659802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:49505)
0926 07:28:56.660039 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:56.660124 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260926 07:28:56.847752 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:56.847370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:46261)
0926 07:28:56.847608 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:56.847659 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
I20260926 07:28:57.448824 17415 raft_consensus.cc:482] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:57.448945 17415 raft_consensus.cc:504] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce29db40e9a47169f9ec63fccd90d1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38273 } } peers { permanent_uuid: "4901232023b24e9c91576f95abce51a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33189 } } peers { permanent_uuid: "055a3aea67e34bafa5dbb2a0a347e50a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260926 07:28:57.449398 17415 leader_election.cc:288] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273), 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189)
W20260926 07:28:57.450137 16985 leader_election.cc:341] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ce29db40e9a47169f9ec63fccd90d1f (127.0.98.2:38273): Not found: Tablet not found: cc8d0eb903444e90a328e4b9dfc90a4e
W20260926 07:28:57.450224 16982 leader_election.cc:341] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4901232023b24e9c91576f95abce51a3 (127.0.98.3:33189): Not found: Tablet not found: cc8d0eb903444e90a328e4b9dfc90a4e
I20260926 07:28:57.450340 16982 leader_election.cc:302] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 055a3aea67e34bafa5dbb2a0a347e50a; no voters: 2ce29db40e9a47169f9ec63fccd90d1f, 4901232023b24e9c91576f95abce51a3
I20260926 07:28:57.450487 17415 raft_consensus.cc:2726] T cc8d0eb903444e90a328e4b9dfc90a4e P 055a3aea67e34bafa5dbb2a0a347e50a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:28:57.512596 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:57.512169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:48219)
0926 07:28:57.512411 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:57.512500 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":84,"threads_started":1}
I20260926 07:28:57.661448 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:57.661065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:39737)
0926 07:28:57.661295 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:57.661348 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260926 07:28:57.832036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 16926
I20260926 07:28:57.837864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 17100
I20260926 07:28:57.843238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 17233
2026-09-26T07:28:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59462 ms)
[----------] 30 tests from SecurityITest (248179 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:28:57 dist-test-slave-g04g krb5kdc[17420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:28:57 dist-test-slave-g04g krb5kdc[17420](info): set up 2 sockets
Sep 26 07:28:57 dist-test-slave-g04g krb5kdc[17420](info): commencing operation
krb5kdc: starting...
W20260926 07:28:58.894915 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:28:58 dist-test-slave-g04g krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:28:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:28:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:28:59.017732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33867
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:59.126367 17433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:59.126614 17433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:59.126652 17433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:59.130713 17433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:28:59.130777 17433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:59.130796 17433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:59.130813 17433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:28:59.130877 17433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:28:59.136351 17433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:59.137411 17433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:59.138405 17433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:28:59.147258 17433 server_base.cc:1034] running on GCE node
W20260926 07:28:59.147236 17441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:59.147275 17439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:59.147425 17438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:59.147990 17433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:59.149026 17433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:59.150187 17433 hybrid_clock.cc:648] HybridClock initialized: now 1790407739150179 us; error 27 us; skew 500 ppm
Sep 26 07:28:59 dist-test-slave-g04g krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:59.152918 17433 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:28:59.153784 17433 webserver.cc:460] Webserver started at http://127.0.98.62:40767/ using document root <none> and password file <none>
I20260926 07:28:59.154325 17433 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:59.154398 17433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:59.154630 17433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:59.156694 17433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "1e1af197447840febc2ed542089b8eeb"
format_stamp: "Formatted at 2026-09-26 07:28:59 on dist-test-slave-g04g"
server_key: "83af752791f58f8184eb8cd359733e15"
server_key_iv: "0ef2b2f1ac60bfe23f24a4c6e34c4d5d"
server_key_version: "encryptionkey@0"
I20260926 07:28:59.157274 17433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e1af197447840febc2ed542089b8eeb"
format_stamp: "Formatted at 2026-09-26 07:28:59 on dist-test-slave-g04g"
server_key: "83af752791f58f8184eb8cd359733e15"
server_key_iv: "0ef2b2f1ac60bfe23f24a4c6e34c4d5d"
server_key_version: "encryptionkey@0"
I20260926 07:28:59.160836 17433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 07:28:59.163221 17448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:59.164453 17433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:59.164578 17433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "1e1af197447840febc2ed542089b8eeb"
format_stamp: "Formatted at 2026-09-26 07:28:59 on dist-test-slave-g04g"
server_key: "83af752791f58f8184eb8cd359733e15"
server_key_iv: "0ef2b2f1ac60bfe23f24a4c6e34c4d5d"
server_key_version: "encryptionkey@0"
I20260926 07:28:59.164680 17433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:59.183537 17433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:59.186859 17433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:59.187095 17433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:59.194053 17433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33867
I20260926 07:28:59.194082 17500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33867 every 8 connection(s)
I20260926 07:28:59.194998 17433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:28:59.197631 17501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:59.203122 17501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb: Bootstrap starting.
I20260926 07:28:59.203485 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 17433
I20260926 07:28:59.203619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:28:59.203958 392 external_mini_cluster.cc:1428] Setting key a9855f0dbbdfa5abaec1a6f97359143f
I20260926 07:28:59.205811 17501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:59.206552 17501 log.cc:824] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:59.208419 17501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb: No bootstrap required, opened a new log
I20260926 07:28:59.211845 17501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1af197447840febc2ed542089b8eeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33867 } }
I20260926 07:28:59.212080 17501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:59.212155 17501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1af197447840febc2ed542089b8eeb, State: Initialized, Role: FOLLOWER
Sep 26 07:28:59 dist-test-slave-g04g krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:28:59.212621 17501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1af197447840febc2ed542089b8eeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33867 } }
I20260926 07:28:59.212790 17501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:28:59.212867 17501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:28:59.212960 17501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:59.213950 17501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1af197447840febc2ed542089b8eeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33867 } }
I20260926 07:28:59.214318 17501 leader_election.cc:302] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e1af197447840febc2ed542089b8eeb; no voters:
I20260926 07:28:59.214679 17501 leader_election.cc:288] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:28:59.214753 17506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:59.214951 17506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [term 1 LEADER]: Becoming Leader. State: Replica: 1e1af197447840febc2ed542089b8eeb, State: Running, Role: LEADER
I20260926 07:28:59.215309 17506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1af197447840febc2ed542089b8eeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33867 } }
I20260926 07:28:59.215965 17501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:28:59.216533 17506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e1af197447840febc2ed542089b8eeb. Latest consensus state: current_term: 1 leader_uuid: "1e1af197447840febc2ed542089b8eeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1af197447840febc2ed542089b8eeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33867 } } }
I20260926 07:28:59.216727 17506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:59.216924 17507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e1af197447840febc2ed542089b8eeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1af197447840febc2ed542089b8eeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33867 } } }
I20260926 07:28:59.217031 17507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb [sys.catalog]: This master's current role is: LEADER
I20260926 07:28:59.217154 17513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:28:59.219152 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.205381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.0.98.62:33867)
0926 07:28:59.205808 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:59.205824 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:59.205859 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:28:59.206543 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:59.206560 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:59.206951 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:59.207338 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:59.208138 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.209178 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.210051 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.210531 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.210702 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.213306 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:59.213319 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:59.213332 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:59.213358 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:59.215345 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:59.215826 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:59.215832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:59.215837 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:59.215952 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:59.216269 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:59.216271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:59.216273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:59.216860 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:59.217720 (+ 860us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:59.217891 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 07:28:59.218179 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":140,"threads_started":1}
I20260926 07:28:59.220359 17513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:28:59.226143 17513 catalog_manager.cc:1347] Generated new cluster ID: 82141c4686fc4880a937b363d42d1206
I20260926 07:28:59.226203 17513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:28:59.244629 17513 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:28:59.245597 17513 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:28:59.249202 17513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb: Generated new TSK 0
I20260926 07:28:59.249868 17513 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:28:59.279857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33867
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:59.389326 17527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:59.389586 17527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:59.389629 17527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:59.393632 17527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:59.393687 17527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:59.393780 17527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:28:59.398972 17527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:59.400048 17527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:59.401047 17527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:59.410629 17532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:59.410672 17535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:59.410677 17533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:59.410883 17527 server_base.cc:1034] running on GCE node
I20260926 07:28:59.411446 17527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:59.412092 17527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:59.413254 17527 hybrid_clock.cc:648] HybridClock initialized: now 1790407739413239 us; error 29 us; skew 500 ppm
Sep 26 07:28:59 dist-test-slave-g04g krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:59.416177 17527 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:28:59.417244 17527 webserver.cc:460] Webserver started at http://127.0.98.1:41921/ using document root <none> and password file <none>
I20260926 07:28:59.417858 17527 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:59.417948 17527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:59.418177 17527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:59.420714 17527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "d6148b021da04fcf87c9d44b754bd161"
format_stamp: "Formatted at 2026-09-26 07:28:59 on dist-test-slave-g04g"
server_key: "6eb0d9bdd913c4e05353e8fc909a1b8a"
server_key_iv: "a1042a0a255b8b166848c3a6fb9bd3fb"
server_key_version: "encryptionkey@0"
I20260926 07:28:59.421293 17527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d6148b021da04fcf87c9d44b754bd161"
format_stamp: "Formatted at 2026-09-26 07:28:59 on dist-test-slave-g04g"
server_key: "6eb0d9bdd913c4e05353e8fc909a1b8a"
server_key_iv: "a1042a0a255b8b166848c3a6fb9bd3fb"
server_key_version: "encryptionkey@0"
I20260926 07:28:59.424947 17527 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 07:28:59.427253 17542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:59.428417 17527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:59.428548 17527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "d6148b021da04fcf87c9d44b754bd161"
format_stamp: "Formatted at 2026-09-26 07:28:59 on dist-test-slave-g04g"
server_key: "6eb0d9bdd913c4e05353e8fc909a1b8a"
server_key_iv: "a1042a0a255b8b166848c3a6fb9bd3fb"
server_key_version: "encryptionkey@0"
I20260926 07:28:59.428653 17527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:59.446408 17527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:59.449250 17527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:59.449476 17527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:59.450055 17527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:59.450881 17527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:59.450953 17527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:59.451021 17527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:59.451081 17527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:59.460106 17527 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42925
I20260926 07:28:59.460141 17655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42925 every 8 connection(s)
I20260926 07:28:59.460989 17527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:28:59.465682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 17527
I20260926 07:28:59.465835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:28:59.466118 392 external_mini_cluster.cc:1428] Setting key 449af397f339eeca7979c2d6bab031a0
Sep 26 07:28:59 dist-test-slave-g04g krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 07:28:59.475297 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.462751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53599 (local address 127.0.98.62:33867)
0926 07:28:59.462886 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:59.462890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:59.463750 (+ 860us) server_negotiation.cc:407] Connection header received
0926 07:28:59.465215 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:59.465218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:59.465265 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:59.465347 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:59.466622 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.467225 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.468891 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.469212 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.469331 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.471746 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:59.471749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:59.471752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:59.471779 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:59.473272 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:59.473788 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:59.473793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:59.473795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:59.473878 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:59.474216 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:59.474220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:59.474222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:59.474407 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:59.474511 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:59.474987 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 07:28:59.475129 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:28:59.476099 17658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.463128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33867 (local address 127.0.98.1:53599)
0926 07:28:59.463550 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:59.463621 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:59.465025 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:59.465521 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:59.465527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:59.465967 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:59.466476 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.466489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.467365 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.467369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.468763 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.468788 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.469373 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.469378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.469551 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.470135 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:59.470151 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:59.471585 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:59.473416 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:59.473422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:59.473436 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:59.473683 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:59.473978 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:59.473981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:59.473983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:59.474122 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:59.474509 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:59.474515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:59.474745 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 07:28:59.474941 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 07:28:59.475274 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20260926 07:28:59.477303 17656 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 07:28:59.477561 17656 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:59.478060 17656 heartbeater.cc:511] Master 127.0.98.62:33867 requested a full tablet report, sending...
I20260926 07:28:59.479694 17465 ts_manager.cc:194] Registered new tserver with Master: d6148b021da04fcf87c9d44b754bd161 (127.0.98.1:42925)
I20260926 07:28:59.480878 17465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:28:59.495962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33867
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:59.606382 17661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:59.606737 17661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:59.606807 17661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:59.610806 17661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:59.610865 17661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:59.610922 17661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:28:59.616099 17661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:59.617096 17661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:59.618079 17661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:59.627529 17666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:59.627539 17669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:59.627573 17667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:59.627589 17661 server_base.cc:1034] running on GCE node
I20260926 07:28:59.628139 17661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:59.628747 17661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:59.629914 17661 hybrid_clock.cc:648] HybridClock initialized: now 1790407739629900 us; error 27 us; skew 500 ppm
Sep 26 07:28:59 dist-test-slave-g04g krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:59.632692 17661 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:28:59.633562 17661 webserver.cc:460] Webserver started at http://127.0.98.2:40159/ using document root <none> and password file <none>
I20260926 07:28:59.634068 17661 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:59.634123 17661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:59.634277 17661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:59.636314 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "be3d2b7f2df346208fb4e25381c02d59"
format_stamp: "Formatted at 2026-09-26 07:28:59 on dist-test-slave-g04g"
server_key: "cd09c8bd42980d4a689d61e96e41552a"
server_key_iv: "a6e26dee7ceba6c5dc23806de606ba98"
server_key_version: "encryptionkey@0"
I20260926 07:28:59.636819 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "be3d2b7f2df346208fb4e25381c02d59"
format_stamp: "Formatted at 2026-09-26 07:28:59 on dist-test-slave-g04g"
server_key: "cd09c8bd42980d4a689d61e96e41552a"
server_key_iv: "a6e26dee7ceba6c5dc23806de606ba98"
server_key_version: "encryptionkey@0"
I20260926 07:28:59.640307 17661 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:28:59.642671 17676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:59.643883 17661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:28:59.643971 17661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "be3d2b7f2df346208fb4e25381c02d59"
format_stamp: "Formatted at 2026-09-26 07:28:59 on dist-test-slave-g04g"
server_key: "cd09c8bd42980d4a689d61e96e41552a"
server_key_iv: "a6e26dee7ceba6c5dc23806de606ba98"
server_key_version: "encryptionkey@0"
I20260926 07:28:59.644050 17661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:59.659641 17661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:59.662273 17661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:59.662417 17661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:59.662987 17661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:59.663776 17661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:59.663828 17661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:59.663866 17661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:59.663888 17661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:59.673002 17661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46055
I20260926 07:28:59.673027 17789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46055 every 8 connection(s)
I20260926 07:28:59.673861 17661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:28:59.676869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 17661
I20260926 07:28:59.677000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:28:59.677302 392 external_mini_cluster.cc:1428] Setting key e723e29768b2276042b74bc3446b7f00
Sep 26 07:28:59 dist-test-slave-g04g krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 07:28:59.688225 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.675595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53061 (local address 127.0.98.62:33867)
0926 07:28:59.675733 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:59.675738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:59.676534 (+ 796us) server_negotiation.cc:407] Connection header received
0926 07:28:59.677963 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:59.677966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:59.678021 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:59.678124 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:59.679449 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.679999 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.681168 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.681607 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.681731 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.684453 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:59.684457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:59.684461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:59.684495 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:59.686128 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:59.686622 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:59.686628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:59.686630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:59.686712 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:59.687103 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:59.687108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:59.687110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:59.687289 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:59.687401 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:59.687939 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 07:28:59.688064 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:28:59.689160 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.675979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33867 (local address 127.0.98.2:53061)
0926 07:28:59.676396 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:59.676437 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:59.677745 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:59.678303 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:59.678310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:59.678760 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:59.679304 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.679319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.680152 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.680156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.681054 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.681064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.681748 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.681751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.681904 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.682539 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:59.682564 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:59.684279 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:59.686249 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:59.686253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:59.686264 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:59.686514 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:59.686802 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:59.686804 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:59.686806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:59.686965 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:59.687400 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:59.687406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:59.687692 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 07:28:59.687916 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 07:28:59.688234 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 07:28:59.690371 17790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33867
I20260926 07:28:59.690619 17790 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:59.691214 17790 heartbeater.cc:511] Master 127.0.98.62:33867 requested a full tablet report, sending...
I20260926 07:28:59.692342 17465 ts_manager.cc:194] Registered new tserver with Master: be3d2b7f2df346208fb4e25381c02d59 (127.0.98.2:46055)
I20260926 07:28:59.692862 17465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:28:59.706823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33867
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:28:59.815299 17795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:28:59.815578 17795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:28:59.815624 17795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:28:59.819813 17795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:28:59.819870 17795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:28:59.819926 17795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:28:59.825011 17795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:28:59.825920 17795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:28:59.826830 17795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:28:59.836030 17800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:59.836107 17795 server_base.cc:1034] running on GCE node
W20260926 07:28:59.836022 17803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:28:59.836030 17801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:28:59.836627 17795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:28:59.837257 17795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:28:59.838443 17795 hybrid_clock.cc:648] HybridClock initialized: now 1790407739838414 us; error 41 us; skew 500 ppm
Sep 26 07:28:59 dist-test-slave-g04g krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:28:59.841130 17795 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:28:59.841922 17795 webserver.cc:460] Webserver started at http://127.0.98.3:34929/ using document root <none> and password file <none>
I20260926 07:28:59.842465 17795 fs_manager.cc:359] Metadata directory not provided
I20260926 07:28:59.842554 17795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:28:59.842777 17795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:28:59.844780 17795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "ffcf6ee52708444f9bb44e1065bf99fc"
format_stamp: "Formatted at 2026-09-26 07:28:59 on dist-test-slave-g04g"
server_key: "74d1d42a12a5548662040d787e90d8c5"
server_key_iv: "f846f2ab9b304596440676ee3c4db90c"
server_key_version: "encryptionkey@0"
I20260926 07:28:59.845345 17795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ffcf6ee52708444f9bb44e1065bf99fc"
format_stamp: "Formatted at 2026-09-26 07:28:59 on dist-test-slave-g04g"
server_key: "74d1d42a12a5548662040d787e90d8c5"
server_key_iv: "f846f2ab9b304596440676ee3c4db90c"
server_key_version: "encryptionkey@0"
I20260926 07:28:59.848966 17795 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:28:59.851279 17810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:59.852447 17795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:28:59.852588 17795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "ffcf6ee52708444f9bb44e1065bf99fc"
format_stamp: "Formatted at 2026-09-26 07:28:59 on dist-test-slave-g04g"
server_key: "74d1d42a12a5548662040d787e90d8c5"
server_key_iv: "f846f2ab9b304596440676ee3c4db90c"
server_key_version: "encryptionkey@0"
I20260926 07:28:59.852695 17795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:28:59.874444 17795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:28:59.877197 17795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:28:59.877393 17795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:28:59.877978 17795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:28:59.878791 17795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:28:59.878862 17795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:59.878931 17795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:28:59.878984 17795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:28:59.887853 17795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45097
I20260926 07:28:59.887873 17923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45097 every 8 connection(s)
I20260926 07:28:59.888696 17795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:28:59.892864 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 17795
I20260926 07:28:59.893000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:28:59.893326 392 external_mini_cluster.cc:1428] Setting key 5efbfe00388f7eac482e275254baf2ef
Sep 26 07:28:59 dist-test-slave-g04g krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:28:59.901589 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.890439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36909 (local address 127.0.98.62:33867)
0926 07:28:59.890572 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:59.890576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:59.891368 (+ 792us) server_negotiation.cc:407] Connection header received
0926 07:28:59.892576 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:59.892579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:59.892627 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:59.892705 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:59.894137 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.894758 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.896067 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.896274 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.896356 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.898645 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:59.898648 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:59.898650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:59.898672 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:59.899929 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:59.900355 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:59.900358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:59.900359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:59.900425 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:59.900700 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:59.900702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:59.900703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:59.900862 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:59.900932 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:59.901391 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 07:28:59.901475 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:28:59.902318 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.890760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33867 (local address 127.0.98.3:36909)
0926 07:28:59.891215 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:59.891259 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:59.892362 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:59.892929 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:59.892939 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:59.893426 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:28:59.893981 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.894000 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.894882 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.894886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.895962 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.895971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.896396 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.896399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.896527 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.897120 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:28:59.897136 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:28:59.898507 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:28:59.900023 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:59.900027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:59.900038 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:59.900266 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:59.900507 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:28:59.900509 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:28:59.900511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:28:59.900626 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:28:59.900951 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:28:59.900956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:28:59.901195 (+ 239us) client_negotiation.cc:769] Sending connection context
0926 07:28:59.901348 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 07:28:59.901568 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":96,"threads_started":1}
I20260926 07:28:59.903333 17924 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33867
I20260926 07:28:59.903527 17924 heartbeater.cc:464] Registering TS with master...
I20260926 07:28:59.904012 17924 heartbeater.cc:511] Master 127.0.98.62:33867 requested a full tablet report, sending...
I20260926 07:28:59.904945 17465 ts_manager.cc:194] Registered new tserver with Master: ffcf6ee52708444f9bb44e1065bf99fc (127.0.98.3:45097)
I20260926 07:28:59.905495 17465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36909
I20260926 07:28:59.906992 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:28:59.915969 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.908924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.0.98.62:33867)
0926 07:28:59.909043 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:59.909047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:59.909215 (+ 168us) server_negotiation.cc:407] Connection header received
0926 07:28:59.909407 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:59.909411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:59.909453 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:59.909543 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:28:59.910185 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.910678 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.911588 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.911821 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.911927 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.913121 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:28:59.913124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:28:59.913127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:28:59.913150 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:28:59.914276 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:59.914676 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:59.914681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:59.914683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:59.914763 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:28:59.915079 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:28:59.915082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:28:59.915083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:28:59.915291 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:28:59.915428 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:59.915663 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 07:28:59.915810 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:28:59.918572 17465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:28:59.921507 17465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:28:59.936854 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.930741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45097 (local address 127.0.0.1:49680)
0926 07:28:59.931642 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:59.931657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:59.931759 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:59.933016 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:59.933018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:59.933039 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:59.933130 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.933135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.934360 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.934364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.935409 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.935420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.936284 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.936287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.936399 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.936437 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 07:28:59.936546 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:28:59.936678 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260926 07:28:59.936980 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.931305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.0.98.1:42925)
0926 07:28:59.931791 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:59.931810 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:59.931848 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:28:59.931947 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:59.931952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:59.932180 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:59.932354 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:59.932638 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.933814 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.935566 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.936400 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.936529 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.936663 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:59.936735 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 07:28:59.936826 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":58,"threads_started":1}
I20260926 07:28:59.937352 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.931325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.0.98.3:45097)
0926 07:28:59.931931 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:59.931945 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:59.931972 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:28:59.932080 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:59.932085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:59.932325 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:59.932495 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:59.933244 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.934235 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.935559 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.936151 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.936281 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.936430 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:59.937125 (+ 695us) server_negotiation.cc:299] Negotiation successful
0926 07:28:59.937224 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":164,"threads_started":1}
I20260926 07:28:59.936854 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.930665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42925 (local address 127.0.0.1:60532)
0926 07:28:59.930978 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:59.931024 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:59.931168 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:59.932338 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:59.932344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:59.932364 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:59.932512 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.932519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.933951 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.933954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.935429 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.935440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.936537 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.936542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.936621 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.936641 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:28:59.936693 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 07:28:59.936744 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":110,"thread_start_us":95,"threads_started":1}
I20260926 07:28:59.937834 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.931576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.0.98.2:46055)
0926 07:28:59.932132 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:59.932142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:59.932895 (+ 753us) server_negotiation.cc:407] Connection header received
0926 07:28:59.933025 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:59.933030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:59.933212 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:59.933347 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:59.933518 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.934466 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.936493 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.937273 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.937389 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.937516 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:59.937610 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 07:28:59.937693 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":103,"threads_started":1}
I20260926 07:28:59.939110 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.931407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46055 (local address 127.0.0.1:32918)
0926 07:28:59.932781 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:59.932792 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:59.932920 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:59.933333 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:59.933336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:59.933352 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:59.933428 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.933433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.934941 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.934944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.936346 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.936357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.937388 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.937392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.937484 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.937505 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:28:59.937584 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:28:59.937650 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260926 07:28:59.940572 17590 tablet_service.cc:1467] Processing CreateTablet for tablet e874dbf2c2b54de3a541f65fea604bec (DEFAULT_TABLE table=test-table [id=90d29048e3314113b3341af735d0f341]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:59.940574 17858 tablet_service.cc:1467] Processing CreateTablet for tablet e874dbf2c2b54de3a541f65fea604bec (DEFAULT_TABLE table=test-table [id=90d29048e3314113b3341af735d0f341]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:59.941517 17590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e874dbf2c2b54de3a541f65fea604bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:59.941525 17858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e874dbf2c2b54de3a541f65fea604bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:59.947371 17941 tablet_bootstrap.cc:492] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc: Bootstrap starting.
I20260926 07:28:59.949998 17941 tablet_bootstrap.cc:654] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:59.949780 17724 tablet_service.cc:1467] Processing CreateTablet for tablet e874dbf2c2b54de3a541f65fea604bec (DEFAULT_TABLE table=test-table [id=90d29048e3314113b3341af735d0f341]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:28:59.950789 17724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e874dbf2c2b54de3a541f65fea604bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:28:59.951437 17941 log.cc:824] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:59.951711 17942 tablet_bootstrap.cc:492] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161: Bootstrap starting.
I20260926 07:28:59.953316 17941 tablet_bootstrap.cc:492] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc: No bootstrap required, opened a new log
I20260926 07:28:59.953526 17941 ts_tablet_manager.cc:1397] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:28:59.953816 17942 tablet_bootstrap.cc:654] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:59.954994 17942 log.cc:824] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:59.956945 17942 tablet_bootstrap.cc:492] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161: No bootstrap required, opened a new log
I20260926 07:28:59.956863 17941 raft_consensus.cc:348] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6148b021da04fcf87c9d44b754bd161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42925 } } peers { permanent_uuid: "ffcf6ee52708444f9bb44e1065bf99fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45097 } } peers { permanent_uuid: "be3d2b7f2df346208fb4e25381c02d59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46055 } }
I20260926 07:28:59.957120 17942 ts_tablet_manager.cc:1397] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:28:59.957204 17941 raft_consensus.cc:374] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:59.957298 17941 raft_consensus.cc:729] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffcf6ee52708444f9bb44e1065bf99fc, State: Initialized, Role: FOLLOWER
I20260926 07:28:59.957823 17945 tablet_bootstrap.cc:492] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59: Bootstrap starting.
I20260926 07:28:59.957787 17941 consensus_queue.cc:260] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6148b021da04fcf87c9d44b754bd161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42925 } } peers { permanent_uuid: "ffcf6ee52708444f9bb44e1065bf99fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45097 } } peers { permanent_uuid: "be3d2b7f2df346208fb4e25381c02d59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46055 } }
I20260926 07:28:59.958585 17924 heartbeater.cc:503] Master 127.0.98.62:33867 was elected leader, sending a full tablet report...
I20260926 07:28:59.958948 17941 ts_tablet_manager.cc:1428] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 07:28:59.960314 17945 tablet_bootstrap.cc:654] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59: Neither blocks nor log segments found. Creating new log.
I20260926 07:28:59.960891 17942 raft_consensus.cc:348] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6148b021da04fcf87c9d44b754bd161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42925 } } peers { permanent_uuid: "ffcf6ee52708444f9bb44e1065bf99fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45097 } } peers { permanent_uuid: "be3d2b7f2df346208fb4e25381c02d59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46055 } }
I20260926 07:28:59.961104 17942 raft_consensus.cc:374] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:59.961124 17945 log.cc:824] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59: Log is configured to *not* fsync() on all Append() calls
I20260926 07:28:59.961167 17942 raft_consensus.cc:729] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6148b021da04fcf87c9d44b754bd161, State: Initialized, Role: FOLLOWER
I20260926 07:28:59.961637 17942 consensus_queue.cc:260] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6148b021da04fcf87c9d44b754bd161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42925 } } peers { permanent_uuid: "ffcf6ee52708444f9bb44e1065bf99fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45097 } } peers { permanent_uuid: "be3d2b7f2df346208fb4e25381c02d59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46055 } }
I20260926 07:28:59.962242 17656 heartbeater.cc:503] Master 127.0.98.62:33867 was elected leader, sending a full tablet report...
I20260926 07:28:59.962584 17942 ts_tablet_manager.cc:1428] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:28:59.963258 17945 tablet_bootstrap.cc:492] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59: No bootstrap required, opened a new log
I20260926 07:28:59.963485 17945 ts_tablet_manager.cc:1397] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260926 07:28:59.966677 17945 raft_consensus.cc:348] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6148b021da04fcf87c9d44b754bd161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42925 } } peers { permanent_uuid: "ffcf6ee52708444f9bb44e1065bf99fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45097 } } peers { permanent_uuid: "be3d2b7f2df346208fb4e25381c02d59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46055 } }
I20260926 07:28:59.966887 17945 raft_consensus.cc:374] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:28:59.966970 17945 raft_consensus.cc:729] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3d2b7f2df346208fb4e25381c02d59, State: Initialized, Role: FOLLOWER
I20260926 07:28:59.967370 17945 consensus_queue.cc:260] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6148b021da04fcf87c9d44b754bd161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42925 } } peers { permanent_uuid: "ffcf6ee52708444f9bb44e1065bf99fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45097 } } peers { permanent_uuid: "be3d2b7f2df346208fb4e25381c02d59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46055 } }
I20260926 07:28:59.967974 17790 heartbeater.cc:503] Master 127.0.98.62:33867 was elected leader, sending a full tablet report...
I20260926 07:28:59.968297 17945 ts_tablet_manager.cc:1428] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260926 07:28:59.987977 17948 raft_consensus.cc:482] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:28:59.988164 17948 raft_consensus.cc:504] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6148b021da04fcf87c9d44b754bd161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42925 } } peers { permanent_uuid: "ffcf6ee52708444f9bb44e1065bf99fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45097 } } peers { permanent_uuid: "be3d2b7f2df346208fb4e25381c02d59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46055 } }
I20260926 07:28:59.989392 17948 leader_election.cc:288] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffcf6ee52708444f9bb44e1065bf99fc (127.0.98.3:45097), be3d2b7f2df346208fb4e25381c02d59 (127.0.98.2:46055)
I20260926 07:28:59.993330 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.989751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41009 (local address 127.0.98.3:45097)
0926 07:28:59.989958 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:59.989963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:59.989980 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:28:59.990116 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:59.990120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:59.990179 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:59.990249 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:59.990586 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.991240 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.992333 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.992768 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.992883 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.992947 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:59.993124 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 07:28:59.993207 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260926 07:28:59.993369 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.990089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58505 (local address 127.0.98.2:46055)
0926 07:28:59.990251 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:28:59.990256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:28:59.990271 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:28:59.990318 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:28:59.990320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:28:59.990361 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:28:59.990438 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:28:59.990753 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.991337 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.992392 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.992782 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.992903 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.992963 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:28:59.993196 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 07:28:59.993262 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:28:59.993353 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.989597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46055 (local address 127.0.98.1:58505)
0926 07:28:59.989972 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:59.989986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:59.990093 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:59.990463 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:59.990465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:59.990480 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:59.990639 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.990646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.991458 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.991461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.992247 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.992257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.992927 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.992930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.993043 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.993060 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 07:28:59.993168 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 07:28:59.993231 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":115,"threads_started":1}
I20260926 07:28:59.993337 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:28:59.989547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45097 (local address 127.0.98.1:41009)
0926 07:28:59.989884 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 07:28:59.989900 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:28:59.989992 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:28:59.990297 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:28:59.990300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:28:59.990317 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:28:59.990456 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.990464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.991355 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.991361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.992204 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:28:59.992212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:28:59.992895 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:28:59.992898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:28:59.992991 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:28:59.993024 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:28:59.993084 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 07:28:59.993158 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260926 07:28:59.994027 17744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e874dbf2c2b54de3a541f65fea604bec" candidate_uuid: "d6148b021da04fcf87c9d44b754bd161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be3d2b7f2df346208fb4e25381c02d59" is_pre_election: true
I20260926 07:28:59.994248 17878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e874dbf2c2b54de3a541f65fea604bec" candidate_uuid: "d6148b021da04fcf87c9d44b754bd161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffcf6ee52708444f9bb44e1065bf99fc" is_pre_election: true
I20260926 07:28:59.994338 17744 raft_consensus.cc:2454] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6148b021da04fcf87c9d44b754bd161 in term 0.
I20260926 07:28:59.994542 17878 raft_consensus.cc:2454] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6148b021da04fcf87c9d44b754bd161 in term 0.
I20260926 07:28:59.994877 17543 leader_election.cc:302] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be3d2b7f2df346208fb4e25381c02d59, d6148b021da04fcf87c9d44b754bd161; no voters:
I20260926 07:28:59.995200 17948 raft_consensus.cc:2781] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:28:59.995288 17948 raft_consensus.cc:482] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:28:59.995321 17948 raft_consensus.cc:3037] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:59.996346 17948 raft_consensus.cc:504] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6148b021da04fcf87c9d44b754bd161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42925 } } peers { permanent_uuid: "ffcf6ee52708444f9bb44e1065bf99fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45097 } } peers { permanent_uuid: "be3d2b7f2df346208fb4e25381c02d59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46055 } }
I20260926 07:28:59.996734 17948 leader_election.cc:288] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [CANDIDATE]: Term 1 election: Requested vote from peers ffcf6ee52708444f9bb44e1065bf99fc (127.0.98.3:45097), be3d2b7f2df346208fb4e25381c02d59 (127.0.98.2:46055)
I20260926 07:28:59.997048 17878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e874dbf2c2b54de3a541f65fea604bec" candidate_uuid: "d6148b021da04fcf87c9d44b754bd161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffcf6ee52708444f9bb44e1065bf99fc"
I20260926 07:28:59.997089 17744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e874dbf2c2b54de3a541f65fea604bec" candidate_uuid: "d6148b021da04fcf87c9d44b754bd161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be3d2b7f2df346208fb4e25381c02d59"
I20260926 07:28:59.997153 17878 raft_consensus.cc:3037] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:59.997219 17744 raft_consensus.cc:3037] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:28:59.998183 17878 raft_consensus.cc:2454] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6148b021da04fcf87c9d44b754bd161 in term 1.
I20260926 07:28:59.998322 17744 raft_consensus.cc:2454] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6148b021da04fcf87c9d44b754bd161 in term 1.
I20260926 07:28:59.998494 17544 leader_election.cc:302] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6148b021da04fcf87c9d44b754bd161, ffcf6ee52708444f9bb44e1065bf99fc; no voters:
I20260926 07:28:59.998651 17948 raft_consensus.cc:2781] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:28:59.998857 17948 raft_consensus.cc:686] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [term 1 LEADER]: Becoming Leader. State: Replica: d6148b021da04fcf87c9d44b754bd161, State: Running, Role: LEADER
I20260926 07:28:59.999215 17948 consensus_queue.cc:237] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6148b021da04fcf87c9d44b754bd161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42925 } } peers { permanent_uuid: "ffcf6ee52708444f9bb44e1065bf99fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45097 } } peers { permanent_uuid: "be3d2b7f2df346208fb4e25381c02d59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46055 } }
I20260926 07:29:00.002559 17465 catalog_manager.cc:5495] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6148b021da04fcf87c9d44b754bd161 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d6148b021da04fcf87c9d44b754bd161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6148b021da04fcf87c9d44b754bd161" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffcf6ee52708444f9bb44e1065bf99fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be3d2b7f2df346208fb4e25381c02d59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46055 } health_report { overall_health: UNKNOWN } } }
I20260926 07:29:00.025547 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.022307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.0.98.1:42925)
0926 07:29:00.022440 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:00.022443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:00.022455 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:29:00.022529 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:00.022531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:00.022583 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:00.022673 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:00.022858 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.023416 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.024253 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.024496 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.024577 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.024829 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:00.025247 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:00.025336 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:00.025388 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 07:29:00.025437 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:29:00.031284 17744 raft_consensus.cc:1261] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59 [term 1 FOLLOWER]: Refusing update from remote peer d6148b021da04fcf87c9d44b754bd161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:00.031301 17878 raft_consensus.cc:1261] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc [term 1 FOLLOWER]: Refusing update from remote peer d6148b021da04fcf87c9d44b754bd161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:00.031985 17952 consensus_queue.cc:1035] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be3d2b7f2df346208fb4e25381c02d59" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:00.032168 17948 consensus_queue.cc:1035] T e874dbf2c2b54de3a541f65fea604bec P d6148b021da04fcf87c9d44b754bd161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffcf6ee52708444f9bb44e1065bf99fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:00.040158 17956 mvcc.cc:205] Tried to move back new op lower bound from 7333510103161196544 to 7333510103039668224. Current Snapshot: MvccSnapshot[applied={T|T < 7333510103161196544}]
I20260926 07:29:00.042622 17957 mvcc.cc:205] Tried to move back new op lower bound from 7333510103161196544 to 7333510103039668224. Current Snapshot: MvccSnapshot[applied={T|T < 7333510103161196544}]
I20260926 07:29:00.047365 17465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:00.047549 17465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:00.049801 17465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e1af197447840febc2ed542089b8eeb: Sending DeleteTablet for 3 replicas of tablet e874dbf2c2b54de3a541f65fea604bec
I20260926 07:29:00.050482 17858 tablet_service.cc:1514] Processing DeleteTablet for tablet e874dbf2c2b54de3a541f65fea604bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49680
I20260926 07:29:00.050706 17590 tablet_service.cc:1514] Processing DeleteTablet for tablet e874dbf2c2b54de3a541f65fea604bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60532
I20260926 07:29:00.050973 17965 tablet_replica.cc:330] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc: stopping tablet replica
I20260926 07:29:00.051146 17724 tablet_service.cc:1514] Processing DeleteTablet for tablet e874dbf2c2b54de3a541f65fea604bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32918
I20260926 07:29:00.051317 17965 raft_consensus.cc:2229] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:00.051501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 17527
I20260926 07:29:00.051600 17965 raft_consensus.cc:2258] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:00.053393 17965 ts_tablet_manager.cc:1905] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:00.053738 17967 tablet_replica.cc:330] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59: stopping tablet replica
I20260926 07:29:00.054021 17967 raft_consensus.cc:2229] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:00.054299 17967 raft_consensus.cc:2258] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:00.055792 17967 ts_tablet_manager.cc:1905] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:00.056174 17965 ts_tablet_manager.cc:1918] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:00.056279 17965 log.cc:1196] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/e874dbf2c2b54de3a541f65fea604bec
I20260926 07:29:00.056594 17965 ts_tablet_manager.cc:1939] T e874dbf2c2b54de3a541f65fea604bec P ffcf6ee52708444f9bb44e1065bf99fc: Deleting consensus metadata
I20260926 07:29:00.057883 17450 catalog_manager.cc:4841] TS ffcf6ee52708444f9bb44e1065bf99fc (127.0.98.3:45097): tablet e874dbf2c2b54de3a541f65fea604bec (table test-table [id=90d29048e3314113b3341af735d0f341]) successfully deleted
W20260926 07:29:00.058377 17449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42925 (error 108)
I20260926 07:29:00.058561 17967 ts_tablet_manager.cc:1918] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:00.058650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 17661
I20260926 07:29:00.058663 17967 log.cc:1196] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/e874dbf2c2b54de3a541f65fea604bec
I20260926 07:29:00.058983 17967 ts_tablet_manager.cc:1939] T e874dbf2c2b54de3a541f65fea604bec P be3d2b7f2df346208fb4e25381c02d59: Deleting consensus metadata
I20260926 07:29:00.060155 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.059784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42925 (local address 127.0.0.1:60554)
0926 07:29:00.059976 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:00.060053 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 07:29:00.060360 17449 catalog_manager.cc:4584] TS d6148b021da04fcf87c9d44b754bd161 (127.0.98.1:42925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e874dbf2c2b54de3a541f65fea604bec: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42925: connect: Connection refused (error 111)
I20260926 07:29:00.064386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 17795
I20260926 07:29:00.064800 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.064448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46055 (local address 127.0.0.1:32934)
0926 07:29:00.064629 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:00.064688 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 07:29:00.069984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 17433
2026-09-26T07:29:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2227 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T07:29:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:00Z Disabled control of system clock
I20260926 07:29:00.101985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44943
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:29:00.213871 17972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:00.214144 17972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:00.214180 17972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:00.218119 17972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:00.218173 17972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:00.218189 17972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:00.218205 17972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:00.218220 17972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:00.222955 17972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44943
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:00.223975 17972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:00.225346 17972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:00.233758 17977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:00.233758 17980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:00.233942 17972 server_base.cc:1034] running on GCE node
W20260926 07:29:00.233750 17978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:00.234549 17972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:00.235823 17972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:00.236992 17972 hybrid_clock.cc:648] HybridClock initialized: now 1790407740236961 us; error 49 us; skew 500 ppm
I20260926 07:29:00.238938 17972 webserver.cc:460] Webserver started at http://127.0.98.62:44123/ using document root <none> and password file <none>
I20260926 07:29:00.239539 17972 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:00.239612 17972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:00.239871 17972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:00.241945 17972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "46656dac9c714266a58057b4fc72051e"
format_stamp: "Formatted at 2026-09-26 07:29:00 on dist-test-slave-g04g"
server_key: "6cfea926a1f3bfc0a1d848dbc299dbe5"
server_key_iv: "a152193ebb69467786bac9f570e5ff02"
server_key_version: "encryptionkey@0"
I20260926 07:29:00.242528 17972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "46656dac9c714266a58057b4fc72051e"
format_stamp: "Formatted at 2026-09-26 07:29:00 on dist-test-slave-g04g"
server_key: "6cfea926a1f3bfc0a1d848dbc299dbe5"
server_key_iv: "a152193ebb69467786bac9f570e5ff02"
server_key_version: "encryptionkey@0"
I20260926 07:29:00.246207 17972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:29:00.248662 17986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:00.249941 17972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 07:29:00.250105 17972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "46656dac9c714266a58057b4fc72051e"
format_stamp: "Formatted at 2026-09-26 07:29:00 on dist-test-slave-g04g"
server_key: "6cfea926a1f3bfc0a1d848dbc299dbe5"
server_key_iv: "a152193ebb69467786bac9f570e5ff02"
server_key_version: "encryptionkey@0"
I20260926 07:29:00.250224 17972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:00.264111 17972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:00.264794 17972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:00.264989 17972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:00.272163 17972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44943
I20260926 07:29:00.272150 18038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44943 every 8 connection(s)
I20260926 07:29:00.273505 17972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:00.276850 18039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:00.277626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 17972
I20260926 07:29:00.277761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:00.278134 392 external_mini_cluster.cc:1428] Setting key 46d4830c8bd995ea8bf262f1e8b3f1cf
I20260926 07:29:00.283715 18039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e: Bootstrap starting.
I20260926 07:29:00.286430 18039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:00.287366 18039 log.cc:824] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:00.287528 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.279553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.0.98.62:44943)
0926 07:29:00.280161 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:00.280174 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:00.280214 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:29:00.280857 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:00.280874 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:00.281265 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:00.281542 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:00.281968 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.282914 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.284218 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.284696 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.284812 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.285738 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:00.285743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:00.285754 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:00.286121 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:00.286173 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:00.286202 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:00.286275 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:00.286466 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 07:29:00.286727 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":137,"threads_started":1}
I20260926 07:29:00.289887 18039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e: No bootstrap required, opened a new log
I20260926 07:29:00.294072 18039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46656dac9c714266a58057b4fc72051e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44943 } }
I20260926 07:29:00.294333 18039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:00.294373 18039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46656dac9c714266a58057b4fc72051e, State: Initialized, Role: FOLLOWER
I20260926 07:29:00.294772 18039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46656dac9c714266a58057b4fc72051e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44943 } }
I20260926 07:29:00.294893 18039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:00.294934 18039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:00.295009 18039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:00.296005 18039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46656dac9c714266a58057b4fc72051e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44943 } }
I20260926 07:29:00.296326 18039 leader_election.cc:302] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46656dac9c714266a58057b4fc72051e; no voters:
I20260926 07:29:00.296618 18039 leader_election.cc:288] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:00.296746 18044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:00.297008 18044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [term 1 LEADER]: Becoming Leader. State: Replica: 46656dac9c714266a58057b4fc72051e, State: Running, Role: LEADER
I20260926 07:29:00.297295 18044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46656dac9c714266a58057b4fc72051e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44943 } }
I20260926 07:29:00.297930 18039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:00.298791 18045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46656dac9c714266a58057b4fc72051e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46656dac9c714266a58057b4fc72051e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44943 } } }
I20260926 07:29:00.298913 18045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:00.299223 18046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46656dac9c714266a58057b4fc72051e. Latest consensus state: current_term: 1 leader_uuid: "46656dac9c714266a58057b4fc72051e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46656dac9c714266a58057b4fc72051e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44943 } } }
I20260926 07:29:00.299324 18046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e [sys.catalog]: This master's current role is: LEADER
W20260926 07:29:00.302806 18060 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:29:00.302876 18060 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:00.302968 18052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:00.303874 18052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:00.309152 18052 catalog_manager.cc:1347] Generated new cluster ID: 0bce555e85ed413a86f4a887a178ec62
I20260926 07:29:00.309218 18052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:00.326292 18052 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:00.327641 18052 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:00.335513 18052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e: Generated new TSK 0
I20260926 07:29:00.336369 18052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:00.339407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44943
--builtin_ntp_servers=127.0.98.20:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:29:00.457509 18063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:00.457815 18063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:00.457893 18063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:00.462216 18063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:00.462306 18063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:00.462392 18063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:29:00.467288 18063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:00.468330 18063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:00.469730 18063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:00.479315 18068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:00.479420 18063 server_base.cc:1034] running on GCE node
W20260926 07:29:00.479339 18069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:00.479310 18071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:00.480042 18063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:00.480695 18063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:00.481858 18063 hybrid_clock.cc:648] HybridClock initialized: now 1790407740481847 us; error 30 us; skew 500 ppm
I20260926 07:29:00.489091 18063 webserver.cc:460] Webserver started at http://127.0.98.1:45153/ using document root <none> and password file <none>
I20260926 07:29:00.489765 18063 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:00.489876 18063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:00.490119 18063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:00.492316 18063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "c674524dfde14cc7a151083dd18b12af"
format_stamp: "Formatted at 2026-09-26 07:29:00 on dist-test-slave-g04g"
server_key: "9fd82600080740354771098508332a30"
server_key_iv: "3a5773b7903f95989901a4b32bedf7b3"
server_key_version: "encryptionkey@0"
I20260926 07:29:00.492904 18063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c674524dfde14cc7a151083dd18b12af"
format_stamp: "Formatted at 2026-09-26 07:29:00 on dist-test-slave-g04g"
server_key: "9fd82600080740354771098508332a30"
server_key_iv: "3a5773b7903f95989901a4b32bedf7b3"
server_key_version: "encryptionkey@0"
I20260926 07:29:00.496670 18063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 07:29:00.499197 18077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:00.500571 18063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:29:00.500710 18063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "c674524dfde14cc7a151083dd18b12af"
format_stamp: "Formatted at 2026-09-26 07:29:00 on dist-test-slave-g04g"
server_key: "9fd82600080740354771098508332a30"
server_key_iv: "3a5773b7903f95989901a4b32bedf7b3"
server_key_version: "encryptionkey@0"
I20260926 07:29:00.500820 18063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:00.513083 18063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:00.513778 18063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:00.513969 18063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:00.514575 18063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:00.515626 18063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:00.515693 18063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:00.515775 18063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:00.515820 18063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:00.525238 18063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32983
I20260926 07:29:00.525262 18190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32983 every 8 connection(s)
I20260926 07:29:00.526192 18063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:29:00.536000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 18063
I20260926 07:29:00.536024 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.528213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46815 (local address 127.0.98.62:44943)
0926 07:29:00.528374 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:00.528378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:00.529472 (+ 1094us) server_negotiation.cc:407] Connection header received
0926 07:29:00.530974 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:00.530977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:00.531025 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:00.531180 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:00.532099 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.532626 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.533918 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.534205 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.534305 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.535468 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:00.535471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:00.535473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:00.535484 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:00.535505 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:00.535513 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:00.535581 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:00.535755 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 07:29:00.535830 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:29:00.536125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:29:00.536409 392 external_mini_cluster.cc:1428] Setting key b5f20c2a222d6a1f6d5b23af2219001a
I20260926 07:29:00.537195 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.528614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44943 (local address 127.0.98.1:46815)
0926 07:29:00.529340 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:00.529383 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:00.530745 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:00.531363 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:00.531370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:00.531462 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:00.531965 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:00.531977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.532769 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.532776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:00.533784 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:00.533794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.534371 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.534374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:00.534517 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.535195 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:29:00.535196 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:00.535234 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:29:00.535301 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:00.535596 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:00.535602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:00.535606 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:29:00.535697 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:29:00.535966 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":132,"threads_started":1}
I20260926 07:29:00.538363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44943
--builtin_ntp_servers=127.0.98.20:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:29:00.538352 18191 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44943
I20260926 07:29:00.538580 18191 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:00.539098 18191 heartbeater.cc:511] Master 127.0.98.62:44943 requested a full tablet report, sending...
I20260926 07:29:00.540480 18003 ts_manager.cc:194] Registered new tserver with Master: c674524dfde14cc7a151083dd18b12af (127.0.98.1:32983)
I20260926 07:29:00.541604 18003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46815
W20260926 07:29:00.661677 18194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:00.661962 18194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:00.662030 18194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:00.666100 18194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:00.666173 18194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:00.666260 18194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:29:00.670957 18194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:00.672067 18194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:00.673482 18194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:00.683310 18200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:00.683295 18202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:00.683290 18199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:00.683574 18194 server_base.cc:1034] running on GCE node
I20260926 07:29:00.683952 18194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:00.684656 18194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:00.685835 18194 hybrid_clock.cc:648] HybridClock initialized: now 1790407740685820 us; error 28 us; skew 500 ppm
I20260926 07:29:00.687701 18194 webserver.cc:460] Webserver started at http://127.0.98.2:38357/ using document root <none> and password file <none>
I20260926 07:29:00.688287 18194 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:00.688380 18194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:00.688608 18194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:00.690650 18194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "0a99ad7b97314b7cb5ef0e036d0430a7"
format_stamp: "Formatted at 2026-09-26 07:29:00 on dist-test-slave-g04g"
server_key: "e9bfb17d8603820a7ca966d8477fdc81"
server_key_iv: "63d490d81bb774aa01dacc68b22bee2a"
server_key_version: "encryptionkey@0"
I20260926 07:29:00.691274 18194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0a99ad7b97314b7cb5ef0e036d0430a7"
format_stamp: "Formatted at 2026-09-26 07:29:00 on dist-test-slave-g04g"
server_key: "e9bfb17d8603820a7ca966d8477fdc81"
server_key_iv: "63d490d81bb774aa01dacc68b22bee2a"
server_key_version: "encryptionkey@0"
I20260926 07:29:00.695010 18194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:29:00.697362 18208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:00.698719 18194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:00.698863 18194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "0a99ad7b97314b7cb5ef0e036d0430a7"
format_stamp: "Formatted at 2026-09-26 07:29:00 on dist-test-slave-g04g"
server_key: "e9bfb17d8603820a7ca966d8477fdc81"
server_key_iv: "63d490d81bb774aa01dacc68b22bee2a"
server_key_version: "encryptionkey@0"
I20260926 07:29:00.698987 18194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:00.712598 18194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:00.713263 18194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:00.713459 18194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:00.714017 18194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:00.714933 18194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:00.715003 18194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:00.715081 18194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:00.715142 18194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:00.723632 18194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37253
I20260926 07:29:00.723655 18321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37253 every 8 connection(s)
I20260926 07:29:00.724494 18194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:29:00.725471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 18194
I20260926 07:29:00.725589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:29:00.725915 392 external_mini_cluster.cc:1428] Setting key c3959b57ac29a82056834cf26d55f6ab
I20260926 07:29:00.727989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44943
--builtin_ntp_servers=127.0.98.20:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:29:00.733830 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.726469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56629 (local address 127.0.98.62:44943)
0926 07:29:00.726614 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:00.726636 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:00.727791 (+ 1155us) server_negotiation.cc:407] Connection header received
0926 07:29:00.729240 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:00.729254 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:00.729305 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:00.729389 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:00.730228 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.730765 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.731941 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.732167 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.732286 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.733344 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:00.733347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:00.733350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:00.733363 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:00.733391 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:00.733399 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:00.733477 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:00.733636 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 07:29:00.733692 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:29:00.734623 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.726910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44943 (local address 127.0.98.2:56629)
0926 07:29:00.727652 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:00.727691 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:00.729039 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:00.729548 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:00.729554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:00.729632 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:00.730088 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:00.730099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.730887 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.730893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:00.731821 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:00.731829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.732320 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.732324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:00.732489 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.733090 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:29:00.733091 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:00.733128 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:29:00.733185 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:00.733479 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:00.733491 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:00.733497 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:29:00.733624 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 07:29:00.733898 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":176,"threads_started":1}
I20260926 07:29:00.735739 18322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44943
I20260926 07:29:00.735989 18322 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:00.736639 18322 heartbeater.cc:511] Master 127.0.98.62:44943 requested a full tablet report, sending...
I20260926 07:29:00.737800 18003 ts_manager.cc:194] Registered new tserver with Master: 0a99ad7b97314b7cb5ef0e036d0430a7 (127.0.98.2:37253)
I20260926 07:29:00.738531 18003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56629
W20260926 07:29:00.840082 18325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:00.840318 18325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:00.840354 18325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:00.844360 18325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:00.844410 18325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:00.844462 18325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:29:00.849037 18325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:00.849983 18325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:00.851327 18325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:00.860751 18331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:00.860737 18333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:00.860805 18325 server_base.cc:1034] running on GCE node
W20260926 07:29:00.860720 18330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:00.861310 18325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:00.861922 18325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:00.863107 18325 hybrid_clock.cc:648] HybridClock initialized: now 1790407740863076 us; error 44 us; skew 500 ppm
I20260926 07:29:00.864886 18325 webserver.cc:460] Webserver started at http://127.0.98.3:44093/ using document root <none> and password file <none>
I20260926 07:29:00.865435 18325 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:00.865496 18325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:00.865707 18325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:00.867753 18325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "10f3a6a660254914b92a38a2842f1c88"
format_stamp: "Formatted at 2026-09-26 07:29:00 on dist-test-slave-g04g"
server_key: "2e331e55dba1f7e792e1732d4f94022c"
server_key_iv: "9b794f0119f91e0f723fd558fa98e17f"
server_key_version: "encryptionkey@0"
I20260926 07:29:00.868281 18325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "10f3a6a660254914b92a38a2842f1c88"
format_stamp: "Formatted at 2026-09-26 07:29:00 on dist-test-slave-g04g"
server_key: "2e331e55dba1f7e792e1732d4f94022c"
server_key_iv: "9b794f0119f91e0f723fd558fa98e17f"
server_key_version: "encryptionkey@0"
I20260926 07:29:00.871773 18325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:29:00.874056 18339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:00.875252 18325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:00.875381 18325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "10f3a6a660254914b92a38a2842f1c88"
format_stamp: "Formatted at 2026-09-26 07:29:00 on dist-test-slave-g04g"
server_key: "2e331e55dba1f7e792e1732d4f94022c"
server_key_iv: "9b794f0119f91e0f723fd558fa98e17f"
server_key_version: "encryptionkey@0"
I20260926 07:29:00.875485 18325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:00.892534 18325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:00.893216 18325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:00.893411 18325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:00.894026 18325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:00.894908 18325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:00.894979 18325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:00.895066 18325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:00.895120 18325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:00.904860 18325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35711
I20260926 07:29:00.904886 18452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35711 every 8 connection(s)
I20260926 07:29:00.905850 18325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:29:00.913959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 18325
I20260926 07:29:00.914078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:29:00.914403 392 external_mini_cluster.cc:1428] Setting key 0419347ff18bddcdb8cb590765be2806
I20260926 07:29:00.915326 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.907636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34239 (local address 127.0.98.62:44943)
0926 07:29:00.907785 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:00.907788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:00.908745 (+ 957us) server_negotiation.cc:407] Connection header received
0926 07:29:00.910219 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:00.910222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:00.910283 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:00.910384 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:00.911462 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.911995 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.913130 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.913370 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.913483 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.914656 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:00.914659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:00.914662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:00.914675 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:00.914712 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:00.914724 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:00.914798 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:00.915112 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 07:29:00.915177 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:29:00.916316 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.907956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44943 (local address 127.0.98.3:34239)
0926 07:29:00.908590 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:00.908630 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:00.909973 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:00.910584 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:00.910596 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:00.910702 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:00.911323 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:00.911337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.912118 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.912122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:00.913004 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:00.913013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.913521 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.913525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:00.913719 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.914384 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:29:00.914385 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:00.914423 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:29:00.914482 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:00.914914 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:00.914924 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:00.914932 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:29:00.915098 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 07:29:00.915429 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260926 07:29:00.917305 18453 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44943
I20260926 07:29:00.917524 18453 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:00.918008 18453 heartbeater.cc:511] Master 127.0.98.62:44943 requested a full tablet report, sending...
I20260926 07:29:00.918932 18003 ts_manager.cc:194] Registered new tserver with Master: 10f3a6a660254914b92a38a2842f1c88 (127.0.98.3:35711)
I20260926 07:29:00.919526 18003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34239
I20260926 07:29:00.928226 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:29:00.933619 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.930320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.0.98.62:44943)
0926 07:29:00.930444 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:00.930447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:00.930591 (+ 144us) server_negotiation.cc:407] Connection header received
0926 07:29:00.930693 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:00.930696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:00.930734 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:00.930793 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:00.930976 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.931537 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.932375 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.932592 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.932672 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.933310 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:00.933312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:00.933314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:00.933324 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:00.933342 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:00.933349 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:00.933406 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:00.933455 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 07:29:00.933513 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:29:00.936230 18003 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:29:00.939324 18003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:00.953682 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.947292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35711 (local address 127.0.0.1:38510)
0926 07:29:00.947746 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:00.947798 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:00.947919 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:00.949061 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:00.949064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:00.949084 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:00.949245 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:00.949250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.950631 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.950635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:00.952194 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:00.952205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.953119 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.953121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:00.953243 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.953296 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 07:29:00.953410 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 07:29:00.953524 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":200,"thread_start_us":66,"threads_started":1}
I20260926 07:29:00.953698 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.947950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.0.98.3:35711)
0926 07:29:00.948523 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:00.948540 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:00.948564 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:29:00.948659 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:00.948668 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:00.948888 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:00.949087 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:00.949357 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.950524 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.952321 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.953015 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.953107 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.953289 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:00.953445 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 07:29:00.953543 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":145,"threads_started":1}
I20260926 07:29:00.954418 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.948155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32983 (local address 127.0.0.1:41336)
0926 07:29:00.948898 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:00.948913 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:00.948992 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:00.949539 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:00.949542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:00.949560 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:00.949641 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:00.949655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.950934 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.950937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:00.952054 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:00.952063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.954040 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.954045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:00.954133 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.954167 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:29:00.954237 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 07:29:00.954297 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260926 07:29:00.954583 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.947434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37253 (local address 127.0.0.1:36736)
0926 07:29:00.948360 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:00.948374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:00.948464 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:00.949007 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:00.949016 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:00.949041 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:00.949242 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:00.949250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.950428 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.950433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:00.952194 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:00.952205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.954201 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.954205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:00.954293 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.954317 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:29:00.954369 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 07:29:00.954449 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":214,"thread_start_us":103,"threads_started":1}
I20260926 07:29:00.954906 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.948538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.0.98.1:32983)
0926 07:29:00.948945 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:00.948957 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:00.948981 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:29:00.949213 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:00.949217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:00.949406 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:00.949548 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:00.949742 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.950788 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.952193 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.952999 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.953111 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.953289 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:00.954660 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0926 07:29:00.954760 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":138,"threads_started":1}
I20260926 07:29:00.956025 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:00.947565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.0.98.2:37253)
0926 07:29:00.948403 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:00.948417 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:00.948445 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:29:00.948578 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:00.948585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:00.948831 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:00.949396 (+ 565us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:00.949532 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.950320 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.953347 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:00.954073 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:00.954848 (+ 775us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:00.955740 (+ 892us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:00.955789 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 07:29:00.955877 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":139,"threads_started":1}
I20260926 07:29:00.956497 18387 tablet_service.cc:1467] Processing CreateTablet for tablet 194c97ee802940a08c957ef81326803f (DEFAULT_TABLE table=test-table [id=0c1c0edc6d03425cbd243bc19e5990ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:00.957424 18387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194c97ee802940a08c957ef81326803f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:00.957504 18125 tablet_service.cc:1467] Processing CreateTablet for tablet 194c97ee802940a08c957ef81326803f (DEFAULT_TABLE table=test-table [id=0c1c0edc6d03425cbd243bc19e5990ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:00.958519 18125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194c97ee802940a08c957ef81326803f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:00.958524 18256 tablet_service.cc:1467] Processing CreateTablet for tablet 194c97ee802940a08c957ef81326803f (DEFAULT_TABLE table=test-table [id=0c1c0edc6d03425cbd243bc19e5990ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:00.959467 18256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 194c97ee802940a08c957ef81326803f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:00.963485 18470 tablet_bootstrap.cc:492] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88: Bootstrap starting.
I20260926 07:29:00.965840 18470 tablet_bootstrap.cc:654] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:00.966576 18470 log.cc:824] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:00.966789 18472 tablet_bootstrap.cc:492] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7: Bootstrap starting.
I20260926 07:29:00.966790 18471 tablet_bootstrap.cc:492] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af: Bootstrap starting.
I20260926 07:29:00.968393 18470 tablet_bootstrap.cc:492] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88: No bootstrap required, opened a new log
I20260926 07:29:00.968592 18470 ts_tablet_manager.cc:1397] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 07:29:00.968899 18472 tablet_bootstrap.cc:654] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:00.968899 18471 tablet_bootstrap.cc:654] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:00.969771 18472 log.cc:824] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:00.970115 18471 log.cc:824] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:00.972060 18472 tablet_bootstrap.cc:492] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7: No bootstrap required, opened a new log
I20260926 07:29:00.972261 18472 ts_tablet_manager.cc:1397] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:29:00.972652 18471 tablet_bootstrap.cc:492] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af: No bootstrap required, opened a new log
I20260926 07:29:00.972581 18470 raft_consensus.cc:348] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3a6a660254914b92a38a2842f1c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35711 } } peers { permanent_uuid: "0a99ad7b97314b7cb5ef0e036d0430a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37253 } } peers { permanent_uuid: "c674524dfde14cc7a151083dd18b12af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32983 } }
I20260926 07:29:00.972826 18471 ts_tablet_manager.cc:1397] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:29:00.972855 18470 raft_consensus.cc:374] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:00.972944 18470 raft_consensus.cc:729] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f3a6a660254914b92a38a2842f1c88, State: Initialized, Role: FOLLOWER
I20260926 07:29:00.973451 18470 consensus_queue.cc:260] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3a6a660254914b92a38a2842f1c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35711 } } peers { permanent_uuid: "0a99ad7b97314b7cb5ef0e036d0430a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37253 } } peers { permanent_uuid: "c674524dfde14cc7a151083dd18b12af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32983 } }
I20260926 07:29:00.974157 18453 heartbeater.cc:503] Master 127.0.98.62:44943 was elected leader, sending a full tablet report...
I20260926 07:29:00.974458 18470 ts_tablet_manager.cc:1428] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:29:00.975803 18472 raft_consensus.cc:348] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3a6a660254914b92a38a2842f1c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35711 } } peers { permanent_uuid: "0a99ad7b97314b7cb5ef0e036d0430a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37253 } } peers { permanent_uuid: "c674524dfde14cc7a151083dd18b12af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32983 } }
I20260926 07:29:00.976019 18472 raft_consensus.cc:374] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:00.976110 18472 raft_consensus.cc:729] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a99ad7b97314b7cb5ef0e036d0430a7, State: Initialized, Role: FOLLOWER
I20260926 07:29:00.976576 18472 consensus_queue.cc:260] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3a6a660254914b92a38a2842f1c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35711 } } peers { permanent_uuid: "0a99ad7b97314b7cb5ef0e036d0430a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37253 } } peers { permanent_uuid: "c674524dfde14cc7a151083dd18b12af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32983 } }
I20260926 07:29:00.977487 18322 heartbeater.cc:503] Master 127.0.98.62:44943 was elected leader, sending a full tablet report...
I20260926 07:29:00.977344 18471 raft_consensus.cc:348] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3a6a660254914b92a38a2842f1c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35711 } } peers { permanent_uuid: "0a99ad7b97314b7cb5ef0e036d0430a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37253 } } peers { permanent_uuid: "c674524dfde14cc7a151083dd18b12af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32983 } }
I20260926 07:29:00.977583 18471 raft_consensus.cc:374] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:00.977676 18471 raft_consensus.cc:729] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c674524dfde14cc7a151083dd18b12af, State: Initialized, Role: FOLLOWER
I20260926 07:29:00.977689 18472 ts_tablet_manager.cc:1428] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 07:29:00.978137 18471 consensus_queue.cc:260] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3a6a660254914b92a38a2842f1c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35711 } } peers { permanent_uuid: "0a99ad7b97314b7cb5ef0e036d0430a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37253 } } peers { permanent_uuid: "c674524dfde14cc7a151083dd18b12af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32983 } }
I20260926 07:29:00.978792 18191 heartbeater.cc:503] Master 127.0.98.62:44943 was elected leader, sending a full tablet report...
I20260926 07:29:00.979370 18471 ts_tablet_manager.cc:1428] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:29:01.000008 18476 raft_consensus.cc:482] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:01.000231 18476 raft_consensus.cc:504] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3a6a660254914b92a38a2842f1c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35711 } } peers { permanent_uuid: "0a99ad7b97314b7cb5ef0e036d0430a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37253 } } peers { permanent_uuid: "c674524dfde14cc7a151083dd18b12af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32983 } }
I20260926 07:29:01.001296 18476 leader_election.cc:288] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a99ad7b97314b7cb5ef0e036d0430a7 (127.0.98.2:37253), c674524dfde14cc7a151083dd18b12af (127.0.98.1:32983)
I20260926 07:29:01.005098 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:01.001441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37253 (local address 127.0.98.3:35147)
0926 07:29:01.001610 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:01.001626 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:01.001722 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:01.002020 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:01.002024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:01.002041 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:01.002149 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:01.002154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:01.002944 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:01.002947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:01.004027 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:01.004037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:01.004671 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:01.004674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:01.004781 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:01.004802 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 07:29:01.004896 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 07:29:01.004970 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 07:29:01.005167 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:01.001548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35147 (local address 127.0.98.2:37253)
0926 07:29:01.001674 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:01.001678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:01.001692 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:29:01.001845 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:01.001847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:01.001888 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:01.001984 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:01.002279 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:01.002831 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:01.004182 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:01.004553 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:01.004671 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:01.004729 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:01.004969 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 07:29:01.005036 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:29:01.005241 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:01.001641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40469 (local address 127.0.98.1:32983)
0926 07:29:01.001755 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:01.001760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:01.001963 (+ 203us) server_negotiation.cc:407] Connection header received
0926 07:29:01.002143 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:01.002149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:01.002203 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:01.002285 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:01.002539 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:01.003179 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:01.004241 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:01.004702 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:01.004814 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:01.004869 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:01.005064 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 07:29:01.005128 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:29:01.005259 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:01.001540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32983 (local address 127.0.98.3:40469)
0926 07:29:01.001876 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:01.001887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:01.001995 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:01.002294 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:01.002296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:01.002311 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:01.002434 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:01.002439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:01.003291 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:01.003294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:01.004122 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:01.004130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:01.004815 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:01.004819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:01.004915 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:01.004938 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:29:01.005047 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:29:01.005112 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":94,"threads_started":1}
I20260926 07:29:01.006167 18276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194c97ee802940a08c957ef81326803f" candidate_uuid: "10f3a6a660254914b92a38a2842f1c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a99ad7b97314b7cb5ef0e036d0430a7" is_pre_election: true
I20260926 07:29:01.006170 18145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194c97ee802940a08c957ef81326803f" candidate_uuid: "10f3a6a660254914b92a38a2842f1c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c674524dfde14cc7a151083dd18b12af" is_pre_election: true
I20260926 07:29:01.006459 18276 raft_consensus.cc:2454] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f3a6a660254914b92a38a2842f1c88 in term 0.
I20260926 07:29:01.006459 18145 raft_consensus.cc:2454] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f3a6a660254914b92a38a2842f1c88 in term 0.
I20260926 07:29:01.006930 18342 leader_election.cc:302] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a99ad7b97314b7cb5ef0e036d0430a7, 10f3a6a660254914b92a38a2842f1c88; no voters:
I20260926 07:29:01.007215 18476 raft_consensus.cc:2781] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:01.007306 18476 raft_consensus.cc:482] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:01.007367 18476 raft_consensus.cc:3037] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:01.008322 18476 raft_consensus.cc:504] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3a6a660254914b92a38a2842f1c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35711 } } peers { permanent_uuid: "0a99ad7b97314b7cb5ef0e036d0430a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37253 } } peers { permanent_uuid: "c674524dfde14cc7a151083dd18b12af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32983 } }
I20260926 07:29:01.008649 18476 leader_election.cc:288] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [CANDIDATE]: Term 1 election: Requested vote from peers 0a99ad7b97314b7cb5ef0e036d0430a7 (127.0.98.2:37253), c674524dfde14cc7a151083dd18b12af (127.0.98.1:32983)
I20260926 07:29:01.008983 18145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194c97ee802940a08c957ef81326803f" candidate_uuid: "10f3a6a660254914b92a38a2842f1c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c674524dfde14cc7a151083dd18b12af"
I20260926 07:29:01.009119 18145 raft_consensus.cc:3037] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:01.010238 18145 raft_consensus.cc:2454] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f3a6a660254914b92a38a2842f1c88 in term 1.
I20260926 07:29:01.010509 18276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "194c97ee802940a08c957ef81326803f" candidate_uuid: "10f3a6a660254914b92a38a2842f1c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a99ad7b97314b7cb5ef0e036d0430a7"
I20260926 07:29:01.010535 18343 leader_election.cc:302] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f3a6a660254914b92a38a2842f1c88, c674524dfde14cc7a151083dd18b12af; no voters:
I20260926 07:29:01.010619 18276 raft_consensus.cc:3037] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:01.010689 18476 raft_consensus.cc:2781] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:01.010929 18476 raft_consensus.cc:686] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [term 1 LEADER]: Becoming Leader. State: Replica: 10f3a6a660254914b92a38a2842f1c88, State: Running, Role: LEADER
I20260926 07:29:01.011296 18476 consensus_queue.cc:237] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3a6a660254914b92a38a2842f1c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35711 } } peers { permanent_uuid: "0a99ad7b97314b7cb5ef0e036d0430a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37253 } } peers { permanent_uuid: "c674524dfde14cc7a151083dd18b12af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32983 } }
I20260926 07:29:01.011909 18276 raft_consensus.cc:2454] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f3a6a660254914b92a38a2842f1c88 in term 1.
I20260926 07:29:01.014356 18002 catalog_manager.cc:5495] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f3a6a660254914b92a38a2842f1c88 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "10f3a6a660254914b92a38a2842f1c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f3a6a660254914b92a38a2842f1c88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a99ad7b97314b7cb5ef0e036d0430a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c674524dfde14cc7a151083dd18b12af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32983 } health_report { overall_health: UNKNOWN } } }
W20260926 07:29:01.027298 18192 tablet.cc:2369] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:29:01.032732 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:01.029438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.0.98.3:35711)
0926 07:29:01.029564 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:01.029567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:01.029600 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:29:01.029666 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:01.029668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:01.029714 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:01.029850 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:01.030082 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:01.030555 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:01.031421 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:01.031626 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:01.031712 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:01.031943 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:01.032367 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:01.032457 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:01.032512 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 07:29:01.032553 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:29:01.037883 18145 raft_consensus.cc:1261] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af [term 1 FOLLOWER]: Refusing update from remote peer 10f3a6a660254914b92a38a2842f1c88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:01.037976 18276 raft_consensus.cc:1261] T 194c97ee802940a08c957ef81326803f P 0a99ad7b97314b7cb5ef0e036d0430a7 [term 1 FOLLOWER]: Refusing update from remote peer 10f3a6a660254914b92a38a2842f1c88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:01.038569 18480 consensus_queue.cc:1035] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c674524dfde14cc7a151083dd18b12af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:01.038743 18476 consensus_queue.cc:1035] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a99ad7b97314b7cb5ef0e036d0430a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:01.044497 18483 mvcc.cc:205] Tried to move back new op lower bound from 7333510107284893696 to 7333510107185295360. Current Snapshot: MvccSnapshot[applied={T|T < 7333510107284893696}]
I20260926 07:29:01.047345 18485 mvcc.cc:205] Tried to move back new op lower bound from 7333510107284893696 to 7333510107185295360. Current Snapshot: MvccSnapshot[applied={T|T < 7333510107284893696}]
I20260926 07:29:01.047640 18484 mvcc.cc:205] Tried to move back new op lower bound from 7333510107284893696 to 7333510107185295360. Current Snapshot: MvccSnapshot[applied={T|T < 7333510107284893696}]
I20260926 07:29:01.052542 18003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:01.052682 18003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:01.054989 18003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46656dac9c714266a58057b4fc72051e: Sending DeleteTablet for 3 replicas of tablet 194c97ee802940a08c957ef81326803f
I20260926 07:29:01.055815 18125 tablet_service.cc:1514] Processing DeleteTablet for tablet 194c97ee802940a08c957ef81326803f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:01 UTC) from {username='slave'} at 127.0.0.1:41336
I20260926 07:29:01.055823 18387 tablet_service.cc:1514] Processing DeleteTablet for tablet 194c97ee802940a08c957ef81326803f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:01 UTC) from {username='slave'} at 127.0.0.1:38510
I20260926 07:29:01.056299 18493 tablet_replica.cc:330] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af: stopping tablet replica
I20260926 07:29:01.056450 18256 tablet_service.cc:1514] Processing DeleteTablet for tablet 194c97ee802940a08c957ef81326803f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:01 UTC) from {username='slave'} at 127.0.0.1:36736
I20260926 07:29:01.056743 18493 raft_consensus.cc:2229] T 194c97ee802940a08c957ef81326803f P c674524dfde14cc7a151083dd18b12af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:01.056754 18494 tablet_replica.cc:330] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88: stopping tablet replica
I20260926 07:29:01.056978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 18063
I20260926 07:29:01.057004 18494 raft_consensus.cc:2229] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:29:01.057372 18494 raft_consensus.cc:2258] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:01.058773 18494 ts_tablet_manager.cc:1905] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:01.061532 18494 ts_tablet_manager.cc:1918] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:01.061630 18494 log.cc:1196] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/194c97ee802940a08c957ef81326803f
I20260926 07:29:01.061930 18494 ts_tablet_manager.cc:1939] T 194c97ee802940a08c957ef81326803f P 10f3a6a660254914b92a38a2842f1c88: Deleting consensus metadata
I20260926 07:29:01.063102 17990 catalog_manager.cc:4841] TS 10f3a6a660254914b92a38a2842f1c88 (127.0.98.3:35711): tablet 194c97ee802940a08c957ef81326803f (table test-table [id=0c1c0edc6d03425cbd243bc19e5990ee]) successfully deleted
W20260926 07:29:01.064373 17990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32983 (error 108)
I20260926 07:29:01.065099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 18194
I20260926 07:29:01.065412 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:01.065095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32983 (local address 127.0.0.1:41350)
0926 07:29:01.065232 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:01.065312 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 07:29:01.066004 17990 catalog_manager.cc:4584] TS c674524dfde14cc7a151083dd18b12af (127.0.98.1:32983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 194c97ee802940a08c957ef81326803f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32983: connect: Connection refused (error 111)
I20260926 07:29:01.072489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 18325
I20260926 07:29:01.073376 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:01.072652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37253 (local address 127.0.0.1:36742)
0926 07:29:01.073220 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:01.073285 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463}
I20260926 07:29:01.078174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 17972
2026-09-26T07:29:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1006 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:01 dist-test-slave-g04g krb5kdc[18499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:01 dist-test-slave-g04g krb5kdc[18499](info): set up 2 sockets
Sep 26 07:29:01 dist-test-slave-g04g krb5kdc[18499](info): commencing operation
krb5kdc: starting...
W20260926 07:29:02.115067 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:02 dist-test-slave-g04g krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:29:02.234853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34245
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:02.343289 18512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:02.343544 18512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:02.343585 18512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:02.347469 18512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:02.347514 18512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:02.347533 18512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:02.347549 18512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:02.347564 18512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:02.352634 18512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:02.353602 18512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:02.354550 18512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:02.362813 18518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:02.363020 18512 server_base.cc:1034] running on GCE node
W20260926 07:29:02.362813 18520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:02.362871 18517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:02.363795 18512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:02.364902 18512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:02.366086 18512 hybrid_clock.cc:648] HybridClock initialized: now 1790407742366054 us; error 48 us; skew 500 ppm
Sep 26 07:29:02 dist-test-slave-g04g krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:02.368999 18512 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:29:02.369980 18512 webserver.cc:460] Webserver started at http://127.0.98.62:34973/ using document root <none> and password file <none>
I20260926 07:29:02.370550 18512 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:02.370640 18512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:02.370873 18512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:02.373034 18512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "aea6ffcd49b647adab98d8b7849e97a2"
format_stamp: "Formatted at 2026-09-26 07:29:02 on dist-test-slave-g04g"
server_key: "0da1ff75b36a147be168ea37a117e809"
server_key_iv: "3b21b790c33ce2daa7adbb2357a45e9c"
server_key_version: "encryptionkey@0"
I20260926 07:29:02.373621 18512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "aea6ffcd49b647adab98d8b7849e97a2"
format_stamp: "Formatted at 2026-09-26 07:29:02 on dist-test-slave-g04g"
server_key: "0da1ff75b36a147be168ea37a117e809"
server_key_iv: "3b21b790c33ce2daa7adbb2357a45e9c"
server_key_version: "encryptionkey@0"
I20260926 07:29:02.377434 18512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:29:02.379933 18527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:02.381188 18512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:02.381327 18512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "aea6ffcd49b647adab98d8b7849e97a2"
format_stamp: "Formatted at 2026-09-26 07:29:02 on dist-test-slave-g04g"
server_key: "0da1ff75b36a147be168ea37a117e809"
server_key_iv: "3b21b790c33ce2daa7adbb2357a45e9c"
server_key_version: "encryptionkey@0"
I20260926 07:29:02.381440 18512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:02.440623 18512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:02.443753 18512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:02.443965 18512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:02.451311 18512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34245
I20260926 07:29:02.451339 18579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34245 every 8 connection(s)
I20260926 07:29:02.452301 18512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:02.455161 18580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:02.460513 18580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2: Bootstrap starting.
I20260926 07:29:02.461833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 18512
I20260926 07:29:02.461972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:02.462283 392 external_mini_cluster.cc:1428] Setting key 278bd55f99403e51cb42c01d8b3dc223
I20260926 07:29:02.463310 18580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:02.464123 18580 log.cc:824] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:02.466024 18580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2: No bootstrap required, opened a new log
I20260926 07:29:02.469627 18580 raft_consensus.cc:348] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea6ffcd49b647adab98d8b7849e97a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } }
I20260926 07:29:02.469874 18580 raft_consensus.cc:374] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:02.469957 18580 raft_consensus.cc:729] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea6ffcd49b647adab98d8b7849e97a2, State: Initialized, Role: FOLLOWER
I20260926 07:29:02.470429 18580 consensus_queue.cc:260] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea6ffcd49b647adab98d8b7849e97a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } }
I20260926 07:29:02.470580 18580 raft_consensus.cc:388] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 26 07:29:02 dist-test-slave-g04g krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:02.470669 18580 raft_consensus.cc:482] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:02.470767 18580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:02.471889 18580 raft_consensus.cc:504] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea6ffcd49b647adab98d8b7849e97a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } }
I20260926 07:29:02.472294 18580 leader_election.cc:302] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aea6ffcd49b647adab98d8b7849e97a2; no voters:
I20260926 07:29:02.472638 18580 leader_election.cc:288] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:02.472745 18585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:02.472941 18585 raft_consensus.cc:686] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [term 1 LEADER]: Becoming Leader. State: Replica: aea6ffcd49b647adab98d8b7849e97a2, State: Running, Role: LEADER
I20260926 07:29:02.473258 18585 consensus_queue.cc:237] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea6ffcd49b647adab98d8b7849e97a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } }
I20260926 07:29:02.473989 18580 sys_catalog.cc:564] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:02.474503 18587 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aea6ffcd49b647adab98d8b7849e97a2. Latest consensus state: current_term: 1 leader_uuid: "aea6ffcd49b647adab98d8b7849e97a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea6ffcd49b647adab98d8b7849e97a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } } }
I20260926 07:29:02.474644 18587 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:02.474896 18586 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aea6ffcd49b647adab98d8b7849e97a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aea6ffcd49b647adab98d8b7849e97a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } } }
I20260926 07:29:02.474996 18586 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:02.475373 18589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:02.476881 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:02.463440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.0.98.62:34245)
0926 07:29:02.463876 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:02.463891 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:02.463925 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:29:02.464639 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:02.464665 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:02.465043 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:02.465377 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:02.466141 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:02.467152 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:02.468008 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:02.468467 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:02.468644 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:02.471351 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:02.471356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:02.471369 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:02.471395 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:02.473456 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:02.473931 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:02.473937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:02.473942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:02.474072 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:02.474377 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:02.474381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:02.474382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:02.474949 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:02.475172 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:02.475521 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 07:29:02.475845 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":302,"thread_start_us":134,"threads_started":1}
I20260926 07:29:02.478397 18589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:02.483549 18589 catalog_manager.cc:1347] Generated new cluster ID: c8c8779f34a54cffb4f234ee7d49b5d4
I20260926 07:29:02.483616 18589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:02.489358 18589 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:02.490170 18589 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:02.498689 18589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2: Generated new TSK 0
I20260926 07:29:02.499382 18589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:29:02.529520 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34245
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:02.642081 18606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:02.642375 18606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:02.642468 18606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:02.646531 18606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:02.646620 18606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:02.646711 18606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:29:02.652064 18606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:02.653128 18606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:02.654155 18606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:02.664254 18612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:02.664239 18614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:02.664291 18606 server_base.cc:1034] running on GCE node
W20260926 07:29:02.664242 18611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:02.664865 18606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:02.665479 18606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:02.666661 18606 hybrid_clock.cc:648] HybridClock initialized: now 1790407742666628 us; error 49 us; skew 500 ppm
Sep 26 07:29:02 dist-test-slave-g04g krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:02.669461 18606 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:29:02.670321 18606 webserver.cc:460] Webserver started at http://127.0.98.1:33023/ using document root <none> and password file <none>
I20260926 07:29:02.670825 18606 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:02.670881 18606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:02.671106 18606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:02.673084 18606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "e1cf199177e443a0bb37daf14e678793"
format_stamp: "Formatted at 2026-09-26 07:29:02 on dist-test-slave-g04g"
server_key: "cd4f85f13760e85b6ec5902ead6ce836"
server_key_iv: "e29c1b4dfa7624c198cdf52b89cf7d5f"
server_key_version: "encryptionkey@0"
I20260926 07:29:02.673702 18606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e1cf199177e443a0bb37daf14e678793"
format_stamp: "Formatted at 2026-09-26 07:29:02 on dist-test-slave-g04g"
server_key: "cd4f85f13760e85b6ec5902ead6ce836"
server_key_iv: "e29c1b4dfa7624c198cdf52b89cf7d5f"
server_key_version: "encryptionkey@0"
I20260926 07:29:02.677659 18606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 07:29:02.680160 18621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:02.681416 18606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:02.681505 18606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "e1cf199177e443a0bb37daf14e678793"
format_stamp: "Formatted at 2026-09-26 07:29:02 on dist-test-slave-g04g"
server_key: "cd4f85f13760e85b6ec5902ead6ce836"
server_key_iv: "e29c1b4dfa7624c198cdf52b89cf7d5f"
server_key_version: "encryptionkey@0"
I20260926 07:29:02.681591 18606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:02.702214 18606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:02.705416 18606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:02.705575 18606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:02.706135 18606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:02.706939 18606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:02.706988 18606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:02.707056 18606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:02.707098 18606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:02.716075 18606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36415
I20260926 07:29:02.716089 18734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36415 every 8 connection(s)
I20260926 07:29:02.717099 18606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:29:02.725853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 18606
I20260926 07:29:02.726012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
Sep 26 07:29:02 dist-test-slave-g04g krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:02.726405 392 external_mini_cluster.cc:1428] Setting key e765afdb1d4ac27144efba048746c21c
I20260926 07:29:02.730207 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:02.718918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35721 (local address 127.0.98.62:34245)
0926 07:29:02.719096 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:02.719100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:02.719962 (+ 862us) server_negotiation.cc:407] Connection header received
0926 07:29:02.721189 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:02.721192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:02.721237 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:02.721316 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:02.722460 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:02.722984 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:02.724113 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:02.724348 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:02.724473 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:02.726778 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:02.726781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:02.726784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:02.726809 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:02.728269 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:02.728750 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:02.728756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:02.728758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:02.728840 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:02.729195 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:02.729200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:02.729202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:02.729373 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:02.729471 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:02.729940 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 07:29:02.730052 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:29:02.730994 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:02.719377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34245 (local address 127.0.98.1:35721)
0926 07:29:02.719804 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:02.719845 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:02.720972 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:02.721461 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:02.721467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:02.721808 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:02.722269 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:02.722281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:02.723156 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:02.723158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:02.723998 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:02.724006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:02.724482 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:02.724485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:02.724616 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:02.725138 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:02.725154 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:02.726626 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:02.728381 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:02.728387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:02.728400 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:02.728648 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:02.728934 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:02.728937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:02.728939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:02.729074 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:02.729481 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:02.729489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:02.729712 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 07:29:02.729900 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 07:29:02.730207 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":119,"threads_started":1}
I20260926 07:29:02.732254 18735 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34245
I20260926 07:29:02.732524 18735 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:02.733122 18735 heartbeater.cc:511] Master 127.0.98.62:34245 requested a full tablet report, sending...
I20260926 07:29:02.734613 18544 ts_manager.cc:194] Registered new tserver with Master: e1cf199177e443a0bb37daf14e678793 (127.0.98.1:36415)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 07:29:02.735806 18544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:29:02.754103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34245
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:02.862447 18740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:02.862701 18740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:02.862741 18740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:02.866670 18740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:02.866730 18740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:02.866787 18740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:29:02.872143 18740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:02.873095 18740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:02.874120 18740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:02.883455 18746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:02.883455 18748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:02.883487 18745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:02.883884 18740 server_base.cc:1034] running on GCE node
I20260926 07:29:02.884219 18740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:02.884805 18740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:02.886013 18740 hybrid_clock.cc:648] HybridClock initialized: now 1790407742885985 us; error 55 us; skew 500 ppm
Sep 26 07:29:02 dist-test-slave-g04g krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:02.888710 18740 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:29:02.889638 18740 webserver.cc:460] Webserver started at http://127.0.98.2:43715/ using document root <none> and password file <none>
I20260926 07:29:02.890166 18740 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:02.890224 18740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:02.890446 18740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:02.892423 18740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3"
format_stamp: "Formatted at 2026-09-26 07:29:02 on dist-test-slave-g04g"
server_key: "ac785fb628bb7f0d435254a235d39563"
server_key_iv: "2e58edd6d3b7b461605d9583fc01a568"
server_key_version: "encryptionkey@0"
I20260926 07:29:02.892964 18740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3"
format_stamp: "Formatted at 2026-09-26 07:29:02 on dist-test-slave-g04g"
server_key: "ac785fb628bb7f0d435254a235d39563"
server_key_iv: "2e58edd6d3b7b461605d9583fc01a568"
server_key_version: "encryptionkey@0"
I20260926 07:29:02.896378 18740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:29:02.898581 18755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:02.899948 18740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 07:29:02.900089 18740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3"
format_stamp: "Formatted at 2026-09-26 07:29:02 on dist-test-slave-g04g"
server_key: "ac785fb628bb7f0d435254a235d39563"
server_key_iv: "2e58edd6d3b7b461605d9583fc01a568"
server_key_version: "encryptionkey@0"
I20260926 07:29:02.900192 18740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:02.925257 18740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:02.928522 18740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:02.928720 18740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:02.929307 18740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:02.930177 18740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:02.930244 18740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:02.930315 18740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:02.930367 18740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:02.940183 18740 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37859
I20260926 07:29:02.940253 18868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37859 every 8 connection(s)
I20260926 07:29:02.941072 18740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:29:02.950328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 18740
I20260926 07:29:02.950528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:29:02.950855 392 external_mini_cluster.cc:1428] Setting key 8652759c0291552769787e881ff9bf49
Sep 26 07:29:02 dist-test-slave-g04g krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:02.955757 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:02.943791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40881 (local address 127.0.98.62:34245)
0926 07:29:02.943935 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:02.943940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:02.944826 (+ 886us) server_negotiation.cc:407] Connection header received
0926 07:29:02.946074 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:02.946077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:02.946129 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:02.946223 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:02.947397 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:02.947928 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:02.949028 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:02.949276 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:02.949363 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:02.951973 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:02.951976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:02.951979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:02.952001 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:02.953619 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:02.954122 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:02.954128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:02.954132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:02.954261 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:02.954680 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:02.954682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:02.954683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:02.954844 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:02.954918 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:02.955552 (+ 634us) server_negotiation.cc:299] Negotiation successful
0926 07:29:02.955644 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:29:02.956957 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:02.944255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34245 (local address 127.0.98.2:40881)
0926 07:29:02.944684 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:02.944725 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:02.945858 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:02.946369 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:02.946376 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:02.946714 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:02.947266 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:02.947281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:02.948044 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:02.948046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:02.948912 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:02.948920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:02.949407 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:02.949410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:02.949548 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:02.950164 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:02.950180 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:02.951823 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:02.953731 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:02.953736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:02.953747 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:02.953983 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:02.954369 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:02.954375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:02.954378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:02.954583 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:02.954948 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:02.954955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:02.955301 (+ 346us) client_negotiation.cc:769] Sending connection context
0926 07:29:02.955515 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 07:29:02.955864 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":121,"threads_started":1}
I20260926 07:29:02.958240 18869 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34245
I20260926 07:29:02.958503 18869 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:02.958969 18869 heartbeater.cc:511] Master 127.0.98.62:34245 requested a full tablet report, sending...
I20260926 07:29:02.960129 18544 ts_manager.cc:194] Registered new tserver with Master: 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 (127.0.98.2:37859)
I20260926 07:29:02.960947 18544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40881
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:29:02.977840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34245
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:03.088882 18874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:03.089170 18874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:03.089294 18874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:03.093357 18874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:03.093429 18874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:03.093536 18874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:29:03.098655 18874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:03.099699 18874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:03.100677 18874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:03.111016 18882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:03.110996 18880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:03.111094 18879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:03.111248 18874 server_base.cc:1034] running on GCE node
I20260926 07:29:03.111625 18874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:03.112195 18874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:03.113345 18874 hybrid_clock.cc:648] HybridClock initialized: now 1790407743113331 us; error 26 us; skew 500 ppm
Sep 26 07:29:03 dist-test-slave-g04g krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:03.115921 18874 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:29:03.116715 18874 webserver.cc:460] Webserver started at http://127.0.98.3:36599/ using document root <none> and password file <none>
I20260926 07:29:03.117200 18874 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:03.117255 18874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:03.117409 18874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:03.119391 18874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "e450745e099540a790512ced6d473ac1"
format_stamp: "Formatted at 2026-09-26 07:29:03 on dist-test-slave-g04g"
server_key: "8a667ab51d02cd9f826523040ebadcb9"
server_key_iv: "2a853ceeb242c81991967c0b6748ecfc"
server_key_version: "encryptionkey@0"
I20260926 07:29:03.119879 18874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e450745e099540a790512ced6d473ac1"
format_stamp: "Formatted at 2026-09-26 07:29:03 on dist-test-slave-g04g"
server_key: "8a667ab51d02cd9f826523040ebadcb9"
server_key_iv: "2a853ceeb242c81991967c0b6748ecfc"
server_key_version: "encryptionkey@0"
I20260926 07:29:03.123273 18874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 07:29:03.125448 18889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:03.126605 18874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:03.126742 18874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "e450745e099540a790512ced6d473ac1"
format_stamp: "Formatted at 2026-09-26 07:29:03 on dist-test-slave-g04g"
server_key: "8a667ab51d02cd9f826523040ebadcb9"
server_key_iv: "2a853ceeb242c81991967c0b6748ecfc"
server_key_version: "encryptionkey@0"
I20260926 07:29:03.126848 18874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:03.144172 18874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:03.146857 18874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:03.147101 18874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:03.147652 18874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:03.148427 18874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:03.148504 18874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:03.148593 18874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:03.148643 18874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:03.157218 18874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45685
I20260926 07:29:03.157244 19002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45685 every 8 connection(s)
I20260926 07:29:03.158056 18874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:29:03.163954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 18874
I20260926 07:29:03.164086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:29:03.164407 392 external_mini_cluster.cc:1428] Setting key a04c509f3728e7b5a84f092e2490f693
Sep 26 07:29:03 dist-test-slave-g04g krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:03.171241 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.159798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42031 (local address 127.0.98.62:34245)
0926 07:29:03.159939 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:03.159944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:03.160761 (+ 817us) server_negotiation.cc:407] Connection header received
0926 07:29:03.161980 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:03.161983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:03.162056 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:03.162136 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:03.163283 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.163808 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.165347 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.165629 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.165726 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.168068 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:03.168071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:03.168073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:03.168095 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:03.169554 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:03.169974 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:03.169976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:03.169978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:03.170051 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:03.170320 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:03.170322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:03.170323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:03.170480 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:03.170553 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:03.170973 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 07:29:03.171125 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:29:03.172024 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.160211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34245 (local address 127.0.98.3:42031)
0926 07:29:03.160620 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:03.160659 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:03.161761 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:03.162292 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:03.162298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:03.162634 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:03.163148 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.163162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.163936 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.163939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.165234 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.165244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.165783 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.165787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.165943 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.166527 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:03.166550 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:03.167925 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:03.169648 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:03.169652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:03.169662 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:03.169894 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:03.170126 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:03.170128 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:03.170129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:03.170246 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:03.170562 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:03.170566 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:03.170781 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 07:29:03.170924 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 07:29:03.171235 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":108,"threads_started":1}
I20260926 07:29:03.173017 19003 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34245
I20260926 07:29:03.173223 19003 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:03.173676 19003 heartbeater.cc:511] Master 127.0.98.62:34245 requested a full tablet report, sending...
I20260926 07:29:03.174633 18544 ts_manager.cc:194] Registered new tserver with Master: e450745e099540a790512ced6d473ac1 (127.0.98.3:45685)
I20260926 07:29:03.175283 18544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42031
I20260926 07:29:03.178073 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:29:03.187457 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.180077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.0.98.62:34245)
0926 07:29:03.180186 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:03.180190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:03.180382 (+ 192us) server_negotiation.cc:407] Connection header received
0926 07:29:03.180469 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:03.180471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:03.180504 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:03.180567 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:03.181221 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.181806 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.182586 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.182806 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.182895 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.184325 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:03.184329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:03.184332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:03.184355 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:03.185842 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:03.186324 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:03.186327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:03.186329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:03.186402 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:03.186632 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:03.186634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:03.186635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:03.186806 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:03.186913 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:03.187192 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 07:29:03.187306 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:29:03.190088 18544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:29:03.193053 18544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:03.207067 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.200718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45685 (local address 127.0.0.1:36810)
0926 07:29:03.201605 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:03.201652 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:03.201788 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:03.202540 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:03.202543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:03.202558 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:03.202800 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.202809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.204320 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.204324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.205446 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.205458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.206455 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.206458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.206578 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.206617 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:29:03.206748 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 07:29:03.206868 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":116,"spinlock_wait_cycles":4096,"thread_start_us":64,"threads_started":1}
I20260926 07:29:03.207098 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.200838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.0.98.3:45685)
0926 07:29:03.201172 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:03.201183 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:03.201928 (+ 745us) server_negotiation.cc:407] Connection header received
0926 07:29:03.202023 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:03.202029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:03.202230 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:03.202377 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:03.202943 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.204168 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.205578 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.206336 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.206444 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.206581 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:03.206805 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 07:29:03.206908 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":152,"threads_started":1}
I20260926 07:29:03.207293 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.200788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37859 (local address 127.0.0.1:42048)
0926 07:29:03.201638 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:03.201653 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:03.201788 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:03.202309 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:03.202319 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:03.202346 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:03.202774 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.202782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.204320 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.204323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.205806 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.205816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.206883 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.206887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.206972 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.206999 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 07:29:03.207067 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 07:29:03.207148 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":177,"spinlock_wait_cycles":6144,"thread_start_us":83,"threads_started":1}
I20260926 07:29:03.207785 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.201404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36415 (local address 127.0.0.1:33652)
0926 07:29:03.202479 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:03.202491 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:03.202598 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:03.203067 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:03.203070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:03.203088 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:03.203237 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.203245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.204327 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.204331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.206125 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.206133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.207380 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.207384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.207475 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.207503 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 07:29:03.207576 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 07:29:03.207653 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":164,"spinlock_wait_cycles":384,"thread_start_us":79,"threads_started":1}
I20260926 07:29:03.207851 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.201519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.0.98.2:37859)
0926 07:29:03.201807 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:03.201820 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:03.201846 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:29:03.201924 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:03.201929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:03.202140 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:03.202322 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:03.202912 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.204145 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.205940 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.206746 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.207390 (+ 644us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.207521 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:03.207581 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:29:03.207669 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":128,"threads_started":1}
I20260926 07:29:03.208431 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.201909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.0.98.1:36415)
0926 07:29:03.202180 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:03.202196 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:03.202590 (+ 394us) server_negotiation.cc:407] Connection header received
0926 07:29:03.202718 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:03.202723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:03.202900 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:03.203087 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:03.203333 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.204138 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.206240 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.207036 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.207166 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.207322 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:03.208183 (+ 861us) server_negotiation.cc:299] Negotiation successful
0926 07:29:03.208276 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260926 07:29:03.210656 18937 tablet_service.cc:1467] Processing CreateTablet for tablet 5779355ff62d4c8496ef64776ec2846f (DEFAULT_TABLE table=test-table [id=1e4839208f9348c5bd2ee7ff39412a48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:03.210711 18803 tablet_service.cc:1467] Processing CreateTablet for tablet 5779355ff62d4c8496ef64776ec2846f (DEFAULT_TABLE table=test-table [id=1e4839208f9348c5bd2ee7ff39412a48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:03.210935 18669 tablet_service.cc:1467] Processing CreateTablet for tablet 5779355ff62d4c8496ef64776ec2846f (DEFAULT_TABLE table=test-table [id=1e4839208f9348c5bd2ee7ff39412a48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:03.211835 18803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5779355ff62d4c8496ef64776ec2846f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:03.211835 18937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5779355ff62d4c8496ef64776ec2846f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:03.211937 18669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5779355ff62d4c8496ef64776ec2846f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:03.218724 19021 tablet_bootstrap.cc:492] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1: Bootstrap starting.
I20260926 07:29:03.219143 19020 tablet_bootstrap.cc:492] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793: Bootstrap starting.
I20260926 07:29:03.219594 19022 tablet_bootstrap.cc:492] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3: Bootstrap starting.
I20260926 07:29:03.221211 19022 tablet_bootstrap.cc:654] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:03.221206 19020 tablet_bootstrap.cc:654] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:03.221206 19021 tablet_bootstrap.cc:654] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:03.222214 19022 log.cc:824] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:03.222445 19020 log.cc:824] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:03.222445 19021 log.cc:824] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:03.224246 19021 tablet_bootstrap.cc:492] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1: No bootstrap required, opened a new log
I20260926 07:29:03.224237 19022 tablet_bootstrap.cc:492] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3: No bootstrap required, opened a new log
I20260926 07:29:03.224259 19020 tablet_bootstrap.cc:492] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793: No bootstrap required, opened a new log
I20260926 07:29:03.224426 19022 ts_tablet_manager.cc:1397] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 07:29:03.224426 19021 ts_tablet_manager.cc:1397] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 07:29:03.224447 19020 ts_tablet_manager.cc:1397] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:29:03.227221 19020 raft_consensus.cc:348] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e450745e099540a790512ced6d473ac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45685 } } peers { permanent_uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37859 } } peers { permanent_uuid: "e1cf199177e443a0bb37daf14e678793" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36415 } }
I20260926 07:29:03.227450 19020 raft_consensus.cc:374] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:03.227521 19020 raft_consensus.cc:729] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cf199177e443a0bb37daf14e678793, State: Initialized, Role: FOLLOWER
I20260926 07:29:03.227981 19020 consensus_queue.cc:260] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e450745e099540a790512ced6d473ac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45685 } } peers { permanent_uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37859 } } peers { permanent_uuid: "e1cf199177e443a0bb37daf14e678793" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36415 } }
I20260926 07:29:03.228207 19022 raft_consensus.cc:348] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e450745e099540a790512ced6d473ac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45685 } } peers { permanent_uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37859 } } peers { permanent_uuid: "e1cf199177e443a0bb37daf14e678793" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36415 } }
I20260926 07:29:03.228215 19021 raft_consensus.cc:348] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e450745e099540a790512ced6d473ac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45685 } } peers { permanent_uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37859 } } peers { permanent_uuid: "e1cf199177e443a0bb37daf14e678793" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36415 } }
I20260926 07:29:03.228464 19021 raft_consensus.cc:374] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:03.228531 19022 raft_consensus.cc:374] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:03.228554 19021 raft_consensus.cc:729] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e450745e099540a790512ced6d473ac1, State: Initialized, Role: FOLLOWER
I20260926 07:29:03.228596 18735 heartbeater.cc:503] Master 127.0.98.62:34245 was elected leader, sending a full tablet report...
I20260926 07:29:03.228615 19022 raft_consensus.cc:729] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e871e9e05ae4fdd9ba44bdcdc4f7aa3, State: Initialized, Role: FOLLOWER
I20260926 07:29:03.229080 19022 consensus_queue.cc:260] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e450745e099540a790512ced6d473ac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45685 } } peers { permanent_uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37859 } } peers { permanent_uuid: "e1cf199177e443a0bb37daf14e678793" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36415 } }
I20260926 07:29:03.229080 19021 consensus_queue.cc:260] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e450745e099540a790512ced6d473ac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45685 } } peers { permanent_uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37859 } } peers { permanent_uuid: "e1cf199177e443a0bb37daf14e678793" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36415 } }
I20260926 07:29:03.230015 19022 ts_tablet_manager.cc:1428] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260926 07:29:03.230057 19021 ts_tablet_manager.cc:1428] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 07:29:03.230583 18869 heartbeater.cc:503] Master 127.0.98.62:34245 was elected leader, sending a full tablet report...
I20260926 07:29:03.231449 19003 heartbeater.cc:503] Master 127.0.98.62:34245 was elected leader, sending a full tablet report...
I20260926 07:29:03.231782 19020 ts_tablet_manager.cc:1428] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 07:29:03.303503 19026 raft_consensus.cc:482] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:03.303702 19026 raft_consensus.cc:504] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e450745e099540a790512ced6d473ac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45685 } } peers { permanent_uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37859 } } peers { permanent_uuid: "e1cf199177e443a0bb37daf14e678793" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36415 } }
I20260926 07:29:03.304965 19026 leader_election.cc:288] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e450745e099540a790512ced6d473ac1 (127.0.98.3:45685), 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 (127.0.98.2:37859)
I20260926 07:29:03.309058 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.305354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36375 (local address 127.0.98.3:45685)
0926 07:29:03.305617 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:03.305623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:03.305718 (+ 95us) server_negotiation.cc:407] Connection header received
0926 07:29:03.305890 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:03.305898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:03.305941 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:03.306024 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:03.306329 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.306846 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.308080 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.308495 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.308617 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.308678 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:03.308864 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 07:29:03.308929 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260926 07:29:03.309090 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.305392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38377 (local address 127.0.98.2:37859)
0926 07:29:03.305516 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:03.305521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:03.305776 (+ 255us) server_negotiation.cc:407] Connection header received
0926 07:29:03.305930 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:03.305933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:03.305981 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:03.306065 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:03.306349 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.306838 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.308138 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.308547 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.308664 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.308729 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:03.308907 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 07:29:03.308970 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 07:29:03.309108 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.305261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45685 (local address 127.0.98.1:36375)
0926 07:29:03.305597 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:03.305622 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:03.305777 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:03.306036 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:03.306039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:03.306054 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:03.306222 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.306227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.306959 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.306964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.307956 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.307965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.308625 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.308629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.308730 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.308753 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 07:29:03.308849 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 07:29:03.308919 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":65,"thread_start_us":102,"threads_started":1}
I20260926 07:29:03.309113 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.305261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37859 (local address 127.0.98.1:38377)
0926 07:29:03.305677 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:03.305688 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:03.305815 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:03.306077 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:03.306079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:03.306095 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:03.306244 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.306249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.306947 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.306952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.308022 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.308031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.308666 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.308669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.308778 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.308800 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:29:03.308892 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:29:03.308959 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":94,"thread_start_us":97,"threads_started":1}
I20260926 07:29:03.309751 18957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5779355ff62d4c8496ef64776ec2846f" candidate_uuid: "e1cf199177e443a0bb37daf14e678793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e450745e099540a790512ced6d473ac1" is_pre_election: true
I20260926 07:29:03.309818 18805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5779355ff62d4c8496ef64776ec2846f" candidate_uuid: "e1cf199177e443a0bb37daf14e678793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3" is_pre_election: true
I20260926 07:29:03.310065 18957 raft_consensus.cc:2454] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1cf199177e443a0bb37daf14e678793 in term 0.
I20260926 07:29:03.310084 18805 raft_consensus.cc:2454] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1cf199177e443a0bb37daf14e678793 in term 0.
I20260926 07:29:03.310542 18622 leader_election.cc:302] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7e871e9e05ae4fdd9ba44bdcdc4f7aa3, e1cf199177e443a0bb37daf14e678793, e450745e099540a790512ced6d473ac1; no voters:
I20260926 07:29:03.310825 19026 raft_consensus.cc:2781] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:03.310920 19026 raft_consensus.cc:482] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:03.310981 19026 raft_consensus.cc:3037] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:03.312014 19026 raft_consensus.cc:504] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e450745e099540a790512ced6d473ac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45685 } } peers { permanent_uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37859 } } peers { permanent_uuid: "e1cf199177e443a0bb37daf14e678793" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36415 } }
I20260926 07:29:03.312354 19026 leader_election.cc:288] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [CANDIDATE]: Term 1 election: Requested vote from peers e450745e099540a790512ced6d473ac1 (127.0.98.3:45685), 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 (127.0.98.2:37859)
I20260926 07:29:03.312693 18805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5779355ff62d4c8496ef64776ec2846f" candidate_uuid: "e1cf199177e443a0bb37daf14e678793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3"
I20260926 07:29:03.312712 18957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5779355ff62d4c8496ef64776ec2846f" candidate_uuid: "e1cf199177e443a0bb37daf14e678793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e450745e099540a790512ced6d473ac1"
I20260926 07:29:03.312819 18805 raft_consensus.cc:3037] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:03.312819 18957 raft_consensus.cc:3037] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:03.314070 18957 raft_consensus.cc:2454] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1cf199177e443a0bb37daf14e678793 in term 1.
I20260926 07:29:03.314070 18805 raft_consensus.cc:2454] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1cf199177e443a0bb37daf14e678793 in term 1.
I20260926 07:29:03.314370 18625 leader_election.cc:302] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1cf199177e443a0bb37daf14e678793, e450745e099540a790512ced6d473ac1; no voters:
I20260926 07:29:03.314555 19026 raft_consensus.cc:2781] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:03.314813 19026 raft_consensus.cc:686] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [term 1 LEADER]: Becoming Leader. State: Replica: e1cf199177e443a0bb37daf14e678793, State: Running, Role: LEADER
I20260926 07:29:03.315176 19026 consensus_queue.cc:237] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e450745e099540a790512ced6d473ac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45685 } } peers { permanent_uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37859 } } peers { permanent_uuid: "e1cf199177e443a0bb37daf14e678793" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36415 } }
I20260926 07:29:03.318212 18544 catalog_manager.cc:5495] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1cf199177e443a0bb37daf14e678793 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e1cf199177e443a0bb37daf14e678793" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e450745e099540a790512ced6d473ac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1cf199177e443a0bb37daf14e678793" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36415 } health_report { overall_health: HEALTHY } } }
I20260926 07:29:03.328049 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.324922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.0.98.1:36415)
0926 07:29:03.325039 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:03.325042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:03.325085 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:29:03.325205 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:03.325207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:03.325256 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:03.325339 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:03.325541 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.326004 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.326769 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.326960 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.327079 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.327370 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:03.327735 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:03.327853 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:03.327925 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 07:29:03.327954 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:29:03.333648 18957 raft_consensus.cc:1261] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1 [term 1 FOLLOWER]: Refusing update from remote peer e1cf199177e443a0bb37daf14e678793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:03.333648 18805 raft_consensus.cc:1261] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 [term 1 FOLLOWER]: Refusing update from remote peer e1cf199177e443a0bb37daf14e678793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:03.334334 19026 consensus_queue.cc:1035] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e450745e099540a790512ced6d473ac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:03.334512 19031 consensus_queue.cc:1035] T 5779355ff62d4c8496ef64776ec2846f P e1cf199177e443a0bb37daf14e678793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e871e9e05ae4fdd9ba44bdcdc4f7aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:03.342589 19035 mvcc.cc:205] Tried to move back new op lower bound from 7333510116687560704 to 7333510116621332480. Current Snapshot: MvccSnapshot[applied={T|T < 7333510116687560704}]
I20260926 07:29:03.345364 19036 mvcc.cc:205] Tried to move back new op lower bound from 7333510116687560704 to 7333510116621332480. Current Snapshot: MvccSnapshot[applied={T|T < 7333510116687560704}]
I20260926 07:29:03.349400 18544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:03.349565 18544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:03.351941 18544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aea6ffcd49b647adab98d8b7849e97a2: Sending DeleteTablet for 3 replicas of tablet 5779355ff62d4c8496ef64776ec2846f
I20260926 07:29:03.352669 18803 tablet_service.cc:1514] Processing DeleteTablet for tablet 5779355ff62d4c8496ef64776ec2846f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42048
I20260926 07:29:03.352720 18937 tablet_service.cc:1514] Processing DeleteTablet for tablet 5779355ff62d4c8496ef64776ec2846f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36810
I20260926 07:29:03.352862 18669 tablet_service.cc:1514] Processing DeleteTablet for tablet 5779355ff62d4c8496ef64776ec2846f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33652
I20260926 07:29:03.353158 19045 tablet_replica.cc:330] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1: stopping tablet replica
I20260926 07:29:03.353235 19044 tablet_replica.cc:330] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3: stopping tablet replica
I20260926 07:29:03.353449 19045 raft_consensus.cc:2229] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:03.353443 19044 raft_consensus.cc:2229] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:03.353708 19045 raft_consensus.cc:2258] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:03.353708 19044 raft_consensus.cc:2258] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:03.353909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 18606
I20260926 07:29:03.355448 19044 ts_tablet_manager.cc:1905] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:03.356696 19045 ts_tablet_manager.cc:1905] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:03.358125 19044 ts_tablet_manager.cc:1918] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:03.358224 19044 log.cc:1196] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/5779355ff62d4c8496ef64776ec2846f
I20260926 07:29:03.358600 19044 ts_tablet_manager.cc:1939] T 5779355ff62d4c8496ef64776ec2846f P 7e871e9e05ae4fdd9ba44bdcdc4f7aa3: Deleting consensus metadata
I20260926 07:29:03.359364 19045 ts_tablet_manager.cc:1918] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:03.359453 19045 log.cc:1196] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/5779355ff62d4c8496ef64776ec2846f
I20260926 07:29:03.359812 19045 ts_tablet_manager.cc:1939] T 5779355ff62d4c8496ef64776ec2846f P e450745e099540a790512ced6d473ac1: Deleting consensus metadata
I20260926 07:29:03.359843 18528 catalog_manager.cc:4841] TS 7e871e9e05ae4fdd9ba44bdcdc4f7aa3 (127.0.98.2:37859): tablet 5779355ff62d4c8496ef64776ec2846f (table test-table [id=1e4839208f9348c5bd2ee7ff39412a48]) successfully deleted
I20260926 07:29:03.360903 18531 catalog_manager.cc:4841] TS e450745e099540a790512ced6d473ac1 (127.0.98.3:45685): tablet 5779355ff62d4c8496ef64776ec2846f (table test-table [id=1e4839208f9348c5bd2ee7ff39412a48]) successfully deleted
W20260926 07:29:03.361181 18531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36415 (error 108)
I20260926 07:29:03.361354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 18740
I20260926 07:29:03.362663 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.362342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36415 (local address 127.0.0.1:33656)
0926 07:29:03.362483 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:03.362564 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 07:29:03.362902 18531 catalog_manager.cc:4584] TS e1cf199177e443a0bb37daf14e678793 (127.0.98.1:36415): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5779355ff62d4c8496ef64776ec2846f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36415: connect: Connection refused (error 111)
I20260926 07:29:03.367168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 18874
I20260926 07:29:03.373183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 18512
2026-09-26T07:29:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2296 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T07:29:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:03Z Disabled control of system clock
I20260926 07:29:03.404644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43053
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:29:03.515749 19051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:03.516012 19051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:03.516047 19051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:03.519972 19051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:03.520023 19051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:03.520038 19051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:03.520056 19051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:03.520071 19051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:03.524593 19051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43053
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:03.525513 19051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:03.526800 19051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:03.534825 19059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:03.534808 19056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:03.534915 19057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:03.535166 19051 server_base.cc:1034] running on GCE node
I20260926 07:29:03.535758 19051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:03.536795 19051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:03.537976 19051 hybrid_clock.cc:648] HybridClock initialized: now 1790407743537954 us; error 32 us; skew 500 ppm
I20260926 07:29:03.539779 19051 webserver.cc:460] Webserver started at http://127.0.98.62:35711/ using document root <none> and password file <none>
I20260926 07:29:03.540333 19051 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:03.540393 19051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:03.540642 19051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:03.542595 19051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "2036b9d5505345c1b250d5f9c0dede45"
format_stamp: "Formatted at 2026-09-26 07:29:03 on dist-test-slave-g04g"
server_key: "76a2802d248ca1e19c0610c82d862ccd"
server_key_iv: "c3aca80dbefc6e76f9daf3fbb93e5fd3"
server_key_version: "encryptionkey@0"
I20260926 07:29:03.543187 19051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "2036b9d5505345c1b250d5f9c0dede45"
format_stamp: "Formatted at 2026-09-26 07:29:03 on dist-test-slave-g04g"
server_key: "76a2802d248ca1e19c0610c82d862ccd"
server_key_iv: "c3aca80dbefc6e76f9daf3fbb93e5fd3"
server_key_version: "encryptionkey@0"
I20260926 07:29:03.546598 19051 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:29:03.548851 19065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:03.550040 19051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 07:29:03.550174 19051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "2036b9d5505345c1b250d5f9c0dede45"
format_stamp: "Formatted at 2026-09-26 07:29:03 on dist-test-slave-g04g"
server_key: "76a2802d248ca1e19c0610c82d862ccd"
server_key_iv: "c3aca80dbefc6e76f9daf3fbb93e5fd3"
server_key_version: "encryptionkey@0"
I20260926 07:29:03.550282 19051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:03.570822 19051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:03.571509 19051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:03.571702 19051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:03.578488 19051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43053
I20260926 07:29:03.578497 19117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43053 every 8 connection(s)
I20260926 07:29:03.579403 19051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:03.580121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 19051
I20260926 07:29:03.580281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:03.580623 392 external_mini_cluster.cc:1428] Setting key 5c88aa070ea68bcbb62c3ae207ac06e7
I20260926 07:29:03.582945 19118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:03.588219 19118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45: Bootstrap starting.
I20260926 07:29:03.589336 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.581988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.0.98.62:43053)
0926 07:29:03.582590 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:03.582607 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:03.582640 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:29:03.583377 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:03.583394 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:03.583760 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:03.584079 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:03.584476 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.585391 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.586239 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.586693 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.586860 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.587716 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:03.587721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:03.587732 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:03.588097 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:03.588150 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:03.588169 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:03.588275 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:03.588465 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 07:29:03.588692 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":133,"threads_started":1}
I20260926 07:29:03.590606 19118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:03.591449 19118 log.cc:824] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:03.593127 19118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45: No bootstrap required, opened a new log
I20260926 07:29:03.597105 19118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2036b9d5505345c1b250d5f9c0dede45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43053 } }
I20260926 07:29:03.597409 19118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:03.597498 19118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2036b9d5505345c1b250d5f9c0dede45, State: Initialized, Role: FOLLOWER
I20260926 07:29:03.597937 19118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2036b9d5505345c1b250d5f9c0dede45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43053 } }
I20260926 07:29:03.598096 19118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:03.598169 19118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:03.598289 19118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:03.599412 19118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2036b9d5505345c1b250d5f9c0dede45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43053 } }
I20260926 07:29:03.599860 19118 leader_election.cc:302] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2036b9d5505345c1b250d5f9c0dede45; no voters:
I20260926 07:29:03.600257 19118 leader_election.cc:288] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:03.600360 19123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:03.600584 19123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [term 1 LEADER]: Becoming Leader. State: Replica: 2036b9d5505345c1b250d5f9c0dede45, State: Running, Role: LEADER
I20260926 07:29:03.600874 19123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2036b9d5505345c1b250d5f9c0dede45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43053 } }
I20260926 07:29:03.601497 19118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:03.602415 19124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2036b9d5505345c1b250d5f9c0dede45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2036b9d5505345c1b250d5f9c0dede45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43053 } } }
I20260926 07:29:03.602556 19124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:03.602924 19132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:03.602900 19125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2036b9d5505345c1b250d5f9c0dede45. Latest consensus state: current_term: 1 leader_uuid: "2036b9d5505345c1b250d5f9c0dede45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2036b9d5505345c1b250d5f9c0dede45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43053 } } }
I20260926 07:29:03.603112 19125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:03.606210 19132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:03.611837 19132 catalog_manager.cc:1347] Generated new cluster ID: e2ab1392f63b432aba71b18930c67f1c
I20260926 07:29:03.611925 19132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:03.637897 19132 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:03.638938 19132 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:03.646355 19132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45: Generated new TSK 0
I20260926 07:29:03.647127 19132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:03.653112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43053
--builtin_ntp_servers=127.0.98.20:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:29:03.774507 19142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:03.774758 19142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:03.774796 19142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:03.778801 19142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:03.778855 19142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:03.778909 19142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:29:03.783609 19142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:03.784632 19142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:03.785943 19142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:03.795549 19148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:03.795581 19147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:03.795696 19142 server_base.cc:1034] running on GCE node
W20260926 07:29:03.795621 19150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:03.796268 19142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:03.796897 19142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:03.798064 19142 hybrid_clock.cc:648] HybridClock initialized: now 1790407743798031 us; error 44 us; skew 500 ppm
I20260926 07:29:03.799866 19142 webserver.cc:460] Webserver started at http://127.0.98.1:46585/ using document root <none> and password file <none>
I20260926 07:29:03.800449 19142 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:03.800513 19142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:03.800720 19142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:03.803117 19142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2"
format_stamp: "Formatted at 2026-09-26 07:29:03 on dist-test-slave-g04g"
server_key: "85865e646f008ea19595c3fbb17d72ff"
server_key_iv: "6d6e908e4b21368a168baaf514d301b1"
server_key_version: "encryptionkey@0"
I20260926 07:29:03.803736 19142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2"
format_stamp: "Formatted at 2026-09-26 07:29:03 on dist-test-slave-g04g"
server_key: "85865e646f008ea19595c3fbb17d72ff"
server_key_iv: "6d6e908e4b21368a168baaf514d301b1"
server_key_version: "encryptionkey@0"
I20260926 07:29:03.807397 19142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 07:29:03.809737 19156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:03.810956 19142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260926 07:29:03.811116 19142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2"
format_stamp: "Formatted at 2026-09-26 07:29:03 on dist-test-slave-g04g"
server_key: "85865e646f008ea19595c3fbb17d72ff"
server_key_iv: "6d6e908e4b21368a168baaf514d301b1"
server_key_version: "encryptionkey@0"
I20260926 07:29:03.811223 19142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:03.834468 19142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:03.835273 19142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:03.835480 19142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:03.836102 19142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:03.837041 19142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:03.837087 19142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:03.837157 19142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:03.837188 19142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:03.846412 19142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34431
I20260926 07:29:03.846444 19269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34431 every 8 connection(s)
I20260926 07:29:03.847301 19142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:29:03.849864 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 19142
I20260926 07:29:03.849985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:29:03.850281 392 external_mini_cluster.cc:1428] Setting key afac744e452aa48bbfbfe9d19b5758d5
I20260926 07:29:03.852320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43053
--builtin_ntp_servers=127.0.98.20:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:29:03.856990 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.848994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39415 (local address 127.0.98.62:43053)
0926 07:29:03.849137 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:03.849140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:03.850300 (+ 1160us) server_negotiation.cc:407] Connection header received
0926 07:29:03.852074 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:03.852077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:03.852132 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:03.852209 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:03.853201 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.853791 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.855010 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.855254 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.855369 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.856460 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:03.856464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:03.856466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:03.856492 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:03.856510 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:03.856518 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:03.856593 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:03.856751 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 07:29:03.856849 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:29:03.857764 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:03.849402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43053 (local address 127.0.98.1:39415)
0926 07:29:03.850151 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:03.850191 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:03.851846 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:03.852396 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:03.852406 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:03.852530 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:03.853019 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.853032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.853915 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.853919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.854884 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:03.854894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:03.855404 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:03.855408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:03.855574 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:03.856212 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:29:03.856213 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:03.856253 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:29:03.856311 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:03.856608 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:03.856619 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:03.856630 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 07:29:03.856750 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 07:29:03.857027 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":156,"threads_started":1}
I20260926 07:29:03.858944 19270 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43053
I20260926 07:29:03.859208 19270 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:03.859700 19270 heartbeater.cc:511] Master 127.0.98.62:43053 requested a full tablet report, sending...
I20260926 07:29:03.861320 19082 ts_manager.cc:194] Registered new tserver with Master: 3dd2bbf29e40414ea7adcf99ed46c5b2 (127.0.98.1:34431)
I20260926 07:29:03.863000 19082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39415
W20260926 07:29:03.965416 19273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:03.965662 19273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:03.965698 19273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:03.969585 19273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:03.969656 19273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:03.969709 19273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:29:03.975222 19273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:03.976643 19273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:03.978492 19273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:03.988106 19281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:03.988106 19279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:03.988179 19278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:03.988302 19273 server_base.cc:1034] running on GCE node
I20260926 07:29:03.988758 19273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:03.989399 19273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:03.990579 19273 hybrid_clock.cc:648] HybridClock initialized: now 1790407743990561 us; error 27 us; skew 500 ppm
I20260926 07:29:03.992409 19273 webserver.cc:460] Webserver started at http://127.0.98.2:36051/ using document root <none> and password file <none>
I20260926 07:29:03.992949 19273 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:03.993036 19273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:03.993263 19273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:03.995255 19273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e"
format_stamp: "Formatted at 2026-09-26 07:29:03 on dist-test-slave-g04g"
server_key: "97abce030166d3716cc89ffd48026d73"
server_key_iv: "babcf80637bd43d8fff93461c49fb74c"
server_key_version: "encryptionkey@0"
I20260926 07:29:03.995788 19273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e"
format_stamp: "Formatted at 2026-09-26 07:29:03 on dist-test-slave-g04g"
server_key: "97abce030166d3716cc89ffd48026d73"
server_key_iv: "babcf80637bd43d8fff93461c49fb74c"
server_key_version: "encryptionkey@0"
I20260926 07:29:03.999296 19273 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 07:29:04.001636 19287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:04.002862 19273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:04.002990 19273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e"
format_stamp: "Formatted at 2026-09-26 07:29:03 on dist-test-slave-g04g"
server_key: "97abce030166d3716cc89ffd48026d73"
server_key_iv: "babcf80637bd43d8fff93461c49fb74c"
server_key_version: "encryptionkey@0"
I20260926 07:29:04.003197 19273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:04.014600 19273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:04.015298 19273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:04.015472 19273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:04.015985 19273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:04.016834 19273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:04.016878 19273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:04.016943 19273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:04.016976 19273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:04.025250 19273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35439
I20260926 07:29:04.025276 19400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35439 every 8 connection(s)
I20260926 07:29:04.026068 19273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:29:04.028365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 19273
I20260926 07:29:04.028457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:29:04.028741 392 external_mini_cluster.cc:1428] Setting key bd81e4292b4cf95b46e2b5d762284759
I20260926 07:29:04.030696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43053
--builtin_ntp_servers=127.0.98.20:46377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:29:04.036224 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.027851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53943 (local address 127.0.98.62:43053)
0926 07:29:04.027986 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:04.027990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:04.029118 (+ 1128us) server_negotiation.cc:407] Connection header received
0926 07:29:04.030617 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:04.030621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:04.030678 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:04.030771 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:04.031779 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.032431 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.034187 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.034401 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.034483 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.035778 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:04.035781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:04.035783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:04.035794 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:04.035812 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:04.035820 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:04.035878 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:04.036071 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 07:29:04.036119 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:29:04.037438 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.028235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43053 (local address 127.0.98.2:53943)
0926 07:29:04.028943 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:04.028991 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:04.030411 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:04.030986 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:04.030993 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:04.031138 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:04.031640 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.031654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.032576 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.032579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.034040 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.034054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.034558 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.034562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.034804 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.035458 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:29:04.035463 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:04.035523 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:29:04.035612 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:04.035907 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:04.035919 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:04.035926 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:29:04.036076 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 07:29:04.036386 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":161,"threads_started":1}
I20260926 07:29:04.038563 19401 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43053
I20260926 07:29:04.038808 19401 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:04.039479 19401 heartbeater.cc:511] Master 127.0.98.62:43053 requested a full tablet report, sending...
I20260926 07:29:04.040584 19082 ts_manager.cc:194] Registered new tserver with Master: bb4171cc7cc0400e8d0738aa3dd8bd4e (127.0.98.2:35439)
I20260926 07:29:04.041173 19082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53943
W20260926 07:29:04.142270 19404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:04.142498 19404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:04.142534 19404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:04.146790 19404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:04.146837 19404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:04.146916 19404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:29:04.151690 19404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:04.152629 19404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:04.153988 19404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:04.163348 19409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:04.163530 19404 server_base.cc:1034] running on GCE node
W20260926 07:29:04.163578 19412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:04.163640 19410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:04.164003 19404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:04.164597 19404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:04.165753 19404 hybrid_clock.cc:648] HybridClock initialized: now 1790407744165739 us; error 26 us; skew 500 ppm
I20260926 07:29:04.167479 19404 webserver.cc:460] Webserver started at http://127.0.98.3:45329/ using document root <none> and password file <none>
I20260926 07:29:04.168048 19404 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:04.168108 19404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:04.168350 19404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:04.170310 19404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "dbae1427159e4df983e07351d45034ee"
format_stamp: "Formatted at 2026-09-26 07:29:04 on dist-test-slave-g04g"
server_key: "8166c848a20ff7836a32a7582f1dffc6"
server_key_iv: "9f73262bcbc55d2140e33ac061c3ba11"
server_key_version: "encryptionkey@0"
I20260926 07:29:04.170845 19404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dbae1427159e4df983e07351d45034ee"
format_stamp: "Formatted at 2026-09-26 07:29:04 on dist-test-slave-g04g"
server_key: "8166c848a20ff7836a32a7582f1dffc6"
server_key_iv: "9f73262bcbc55d2140e33ac061c3ba11"
server_key_version: "encryptionkey@0"
I20260926 07:29:04.174299 19404 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 07:29:04.176602 19418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:04.177790 19404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:29:04.177927 19404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "dbae1427159e4df983e07351d45034ee"
format_stamp: "Formatted at 2026-09-26 07:29:04 on dist-test-slave-g04g"
server_key: "8166c848a20ff7836a32a7582f1dffc6"
server_key_iv: "9f73262bcbc55d2140e33ac061c3ba11"
server_key_version: "encryptionkey@0"
I20260926 07:29:04.178036 19404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:04.196034 19404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:04.196653 19404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:04.196838 19404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:04.197391 19404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:04.198282 19404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:04.198350 19404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:04.198416 19404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:04.198469 19404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:04.207647 19404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34487
I20260926 07:29:04.207684 19531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34487 every 8 connection(s)
I20260926 07:29:04.208552 19404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:29:04.217010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 19404
I20260926 07:29:04.217147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:29:04.217420 392 external_mini_cluster.cc:1428] Setting key ab4ce2628825dda940188d720537d5ec
I20260926 07:29:04.217799 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.210348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39973 (local address 127.0.98.62:43053)
0926 07:29:04.210513 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:04.210518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:04.211585 (+ 1067us) server_negotiation.cc:407] Connection header received
0926 07:29:04.213023 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:04.213026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:04.213073 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:04.213156 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:04.214013 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.214616 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.215849 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.216059 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.216140 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.217292 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:04.217295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:04.217297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:04.217310 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:04.217328 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:04.217336 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:04.217414 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:04.217589 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 07:29:04.217649 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:29:04.218782 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.210781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43053 (local address 127.0.98.3:39973)
0926 07:29:04.211444 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:04.211484 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:04.212835 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:04.213316 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:04.213322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:04.213404 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:04.213895 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.213906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.214716 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.214719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.215748 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.215757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.216191 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.216193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.216332 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.216988 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:29:04.216990 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:04.217029 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:29:04.217093 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:04.217419 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:04.217430 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:04.217436 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:29:04.217564 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 07:29:04.217826 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":106,"threads_started":1}
I20260926 07:29:04.219902 19532 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43053
I20260926 07:29:04.220130 19532 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:04.220614 19532 heartbeater.cc:511] Master 127.0.98.62:43053 requested a full tablet report, sending...
I20260926 07:29:04.221566 19081 ts_manager.cc:194] Registered new tserver with Master: dbae1427159e4df983e07351d45034ee (127.0.98.3:34487)
I20260926 07:29:04.222106 19081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39973
I20260926 07:29:04.231287 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:29:04.236508 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.233162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.0.98.62:43053)
0926 07:29:04.233281 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:04.233284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:04.233446 (+ 162us) server_negotiation.cc:407] Connection header received
0926 07:29:04.233539 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:04.233541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:04.233576 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:04.233641 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:04.233879 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.234364 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.235138 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.235335 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.235409 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.236118 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:04.236121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:04.236123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:04.236133 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:04.236151 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:04.236158 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:04.236255 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:04.236380 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 07:29:04.236409 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:29:04.239291 19081 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:29:04.242192 19081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:04.256204 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.250153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34431 (local address 127.0.0.1:41530)
0926 07:29:04.250953 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:04.250968 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:04.251085 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:04.252058 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:04.252060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:04.252076 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:04.252166 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.252172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.253103 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.253109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.254544 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.254554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.255627 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.255633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.255742 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.255778 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:29:04.255891 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 07:29:04.256006 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":53,"thread_start_us":98,"threads_started":1}
I20260926 07:29:04.256228 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.250561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.0.98.1:34431)
0926 07:29:04.251629 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:04.251640 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:04.251662 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:29:04.251738 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:04.251743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:04.251931 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:04.252081 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:04.252276 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.252988 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.254677 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.255498 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.255604 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.255789 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:04.255935 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 07:29:04.256050 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":131,"threads_started":1}
I20260926 07:29:04.256440 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.250169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.0.98.2:35439)
0926 07:29:04.250790 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:04.250802 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:04.250831 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:29:04.251085 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:04.251094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:04.251322 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:04.251479 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:04.251881 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.253013 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.254911 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.255577 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.255692 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.255871 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:04.256158 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 07:29:04.256260 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":159,"threads_started":1}
I20260926 07:29:04.256507 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.250762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34487 (local address 127.0.0.1:59712)
0926 07:29:04.251290 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:04.251302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:04.251389 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:04.252240 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:04.252243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:04.252260 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:04.252338 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.252343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.253815 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.253819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.255074 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.255083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.256125 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.256130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.256219 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.256255 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:29:04.256308 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 07:29:04.256379 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":7,"thread_start_us":82,"threads_started":1}
I20260926 07:29:04.256871 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.250052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35439 (local address 127.0.0.1:55208)
0926 07:29:04.250678 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:04.250727 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:04.250847 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:04.251584 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:04.251590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:04.251612 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:04.251758 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.251765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.253150 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.253153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.254745 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.254754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.255930 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.255933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.256011 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.256031 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:29:04.256641 (+ 610us) client_negotiation.cc:240] Negotiation successful
0926 07:29:04.256725 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":46,"thread_start_us":68,"threads_started":1}
I20260926 07:29:04.257282 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.251230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.0.98.3:34487)
0926 07:29:04.251770 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:04.251778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:04.251796 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:29:04.251887 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:04.251892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:04.252071 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:04.252469 (+ 398us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:04.252591 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.253660 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.255196 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.255976 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.256828 (+ 852us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.257005 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:04.257054 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 07:29:04.257137 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":142,"threads_started":1}
I20260926 07:29:04.259625 19466 tablet_service.cc:1467] Processing CreateTablet for tablet 8328255c8189466b915ac3e688104398 (DEFAULT_TABLE table=test-table [id=9dcf91072e19420686c3f5c30577e2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:04.259656 19335 tablet_service.cc:1467] Processing CreateTablet for tablet 8328255c8189466b915ac3e688104398 (DEFAULT_TABLE table=test-table [id=9dcf91072e19420686c3f5c30577e2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:04.259656 19204 tablet_service.cc:1467] Processing CreateTablet for tablet 8328255c8189466b915ac3e688104398 (DEFAULT_TABLE table=test-table [id=9dcf91072e19420686c3f5c30577e2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:04.260484 19466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8328255c8189466b915ac3e688104398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:04.260610 19204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8328255c8189466b915ac3e688104398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:04.260617 19335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8328255c8189466b915ac3e688104398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:04.267095 19549 tablet_bootstrap.cc:492] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee: Bootstrap starting.
I20260926 07:29:04.267843 19550 tablet_bootstrap.cc:492] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2: Bootstrap starting.
I20260926 07:29:04.269151 19549 tablet_bootstrap.cc:654] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:04.269522 19550 tablet_bootstrap.cc:654] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:04.269923 19549 log.cc:824] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:04.270407 19550 log.cc:824] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:04.270560 19551 tablet_bootstrap.cc:492] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e: Bootstrap starting.
I20260926 07:29:04.272094 19549 tablet_bootstrap.cc:492] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee: No bootstrap required, opened a new log
I20260926 07:29:04.272291 19549 ts_tablet_manager.cc:1397] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:29:04.272449 19550 tablet_bootstrap.cc:492] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2: No bootstrap required, opened a new log
I20260926 07:29:04.272635 19550 ts_tablet_manager.cc:1397] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 07:29:04.272660 19551 tablet_bootstrap.cc:654] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:04.273306 19551 log.cc:824] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:04.274860 19551 tablet_bootstrap.cc:492] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e: No bootstrap required, opened a new log
I20260926 07:29:04.275097 19551 ts_tablet_manager.cc:1397] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:29:04.276760 19550 raft_consensus.cc:348] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35439 } } peers { permanent_uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "dbae1427159e4df983e07351d45034ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34487 } }
I20260926 07:29:04.276760 19549 raft_consensus.cc:348] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35439 } } peers { permanent_uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "dbae1427159e4df983e07351d45034ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34487 } }
I20260926 07:29:04.277029 19549 raft_consensus.cc:374] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:04.277038 19550 raft_consensus.cc:374] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:04.277097 19550 raft_consensus.cc:729] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd2bbf29e40414ea7adcf99ed46c5b2, State: Initialized, Role: FOLLOWER
I20260926 07:29:04.277101 19549 raft_consensus.cc:729] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbae1427159e4df983e07351d45034ee, State: Initialized, Role: FOLLOWER
I20260926 07:29:04.277560 19550 consensus_queue.cc:260] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35439 } } peers { permanent_uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "dbae1427159e4df983e07351d45034ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34487 } }
I20260926 07:29:04.277560 19549 consensus_queue.cc:260] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35439 } } peers { permanent_uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "dbae1427159e4df983e07351d45034ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34487 } }
I20260926 07:29:04.278223 19270 heartbeater.cc:503] Master 127.0.98.62:43053 was elected leader, sending a full tablet report...
I20260926 07:29:04.278481 19532 heartbeater.cc:503] Master 127.0.98.62:43053 was elected leader, sending a full tablet report...
I20260926 07:29:04.278563 19550 ts_tablet_manager.cc:1428] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260926 07:29:04.279577 19549 ts_tablet_manager.cc:1428] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 07:29:04.281982 19551 raft_consensus.cc:348] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35439 } } peers { permanent_uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "dbae1427159e4df983e07351d45034ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34487 } }
I20260926 07:29:04.282279 19551 raft_consensus.cc:374] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:04.282348 19551 raft_consensus.cc:729] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb4171cc7cc0400e8d0738aa3dd8bd4e, State: Initialized, Role: FOLLOWER
I20260926 07:29:04.282825 19551 consensus_queue.cc:260] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35439 } } peers { permanent_uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "dbae1427159e4df983e07351d45034ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34487 } }
I20260926 07:29:04.283437 19401 heartbeater.cc:503] Master 127.0.98.62:43053 was elected leader, sending a full tablet report...
I20260926 07:29:04.283645 19551 ts_tablet_manager.cc:1428] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
W20260926 07:29:04.348441 19271 tablet.cc:2369] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:29:04.459694 19533 tablet.cc:2369] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:29:04.485420 19557 raft_consensus.cc:482] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:04.485666 19557 raft_consensus.cc:504] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35439 } } peers { permanent_uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "dbae1427159e4df983e07351d45034ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34487 } }
I20260926 07:29:04.486883 19557 leader_election.cc:288] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dd2bbf29e40414ea7adcf99ed46c5b2 (127.0.98.1:34431), dbae1427159e4df983e07351d45034ee (127.0.98.3:34487)
I20260926 07:29:04.490666 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.487128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34431 (local address 127.0.98.2:54529)
0926 07:29:04.487350 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:04.487367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:04.487461 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:04.487733 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:04.487736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:04.487754 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:04.487874 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.487880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.488624 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.488627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.489430 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.489439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.490219 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.490222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.490337 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.490367 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:29:04.490446 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:29:04.490517 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":1}
I20260926 07:29:04.490751 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.487247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54529 (local address 127.0.98.1:34431)
0926 07:29:04.487451 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:04.487456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:04.487476 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:29:04.487572 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:04.487574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:04.487623 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:04.487720 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:04.487998 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.488504 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.489556 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.490060 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.490174 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.490236 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:04.490561 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 07:29:04.490631 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:29:04.490983 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.487247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48977 (local address 127.0.98.3:34487)
0926 07:29:04.487395 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:04.487401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:04.487686 (+ 285us) server_negotiation.cc:407] Connection header received
0926 07:29:04.487824 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:04.487828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:04.487873 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:04.487958 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:04.488218 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.488700 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.489723 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.490301 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.490410 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.490484 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:04.490734 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 07:29:04.490830 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:29:04.491423 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.487128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34487 (local address 127.0.98.2:48977)
0926 07:29:04.487598 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:04.487608 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:04.487716 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:04.487968 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:04.487971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:04.487985 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:04.488112 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.488117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.488808 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.488814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.489607 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:04.489615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.490490 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.490494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:04.490577 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.490597 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:29:04.491168 (+ 571us) client_negotiation.cc:240] Negotiation successful
0926 07:29:04.491257 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":65,"thread_start_us":142,"threads_started":1}
I20260926 07:29:04.491627 19224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8328255c8189466b915ac3e688104398" candidate_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2" is_pre_election: true
I20260926 07:29:04.492028 19224 raft_consensus.cc:2454] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb4171cc7cc0400e8d0738aa3dd8bd4e in term 0.
I20260926 07:29:04.492168 19486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8328255c8189466b915ac3e688104398" candidate_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbae1427159e4df983e07351d45034ee" is_pre_election: true
I20260926 07:29:04.492511 19486 raft_consensus.cc:2454] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb4171cc7cc0400e8d0738aa3dd8bd4e in term 0.
I20260926 07:29:04.492606 19289 leader_election.cc:302] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd2bbf29e40414ea7adcf99ed46c5b2, bb4171cc7cc0400e8d0738aa3dd8bd4e; no voters:
I20260926 07:29:04.492885 19557 raft_consensus.cc:2781] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:04.492970 19557 raft_consensus.cc:482] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:04.493026 19557 raft_consensus.cc:3037] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:04.494522 19557 raft_consensus.cc:504] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35439 } } peers { permanent_uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "dbae1427159e4df983e07351d45034ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34487 } }
I20260926 07:29:04.494956 19557 leader_election.cc:288] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [CANDIDATE]: Term 1 election: Requested vote from peers 3dd2bbf29e40414ea7adcf99ed46c5b2 (127.0.98.1:34431), dbae1427159e4df983e07351d45034ee (127.0.98.3:34487)
I20260926 07:29:04.495330 19224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8328255c8189466b915ac3e688104398" candidate_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2"
I20260926 07:29:04.495330 19486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8328255c8189466b915ac3e688104398" candidate_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbae1427159e4df983e07351d45034ee"
I20260926 07:29:04.495447 19224 raft_consensus.cc:3037] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:04.495448 19486 raft_consensus.cc:3037] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:04.496694 19486 raft_consensus.cc:2454] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb4171cc7cc0400e8d0738aa3dd8bd4e in term 1.
I20260926 07:29:04.496699 19224 raft_consensus.cc:2454] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb4171cc7cc0400e8d0738aa3dd8bd4e in term 1.
I20260926 07:29:04.497038 19290 leader_election.cc:302] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb4171cc7cc0400e8d0738aa3dd8bd4e, dbae1427159e4df983e07351d45034ee; no voters:
I20260926 07:29:04.497295 19557 raft_consensus.cc:2781] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:04.497516 19557 raft_consensus.cc:686] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [term 1 LEADER]: Becoming Leader. State: Replica: bb4171cc7cc0400e8d0738aa3dd8bd4e, State: Running, Role: LEADER
I20260926 07:29:04.497998 19557 consensus_queue.cc:237] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35439 } } peers { permanent_uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "dbae1427159e4df983e07351d45034ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34487 } }
I20260926 07:29:04.501341 19082 catalog_manager.cc:5495] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e reported cstate change: term changed from 0 to 1, leader changed from <none> to bb4171cc7cc0400e8d0738aa3dd8bd4e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb4171cc7cc0400e8d0738aa3dd8bd4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbae1427159e4df983e07351d45034ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34487 } health_report { overall_health: UNKNOWN } } }
W20260926 07:29:04.527227 19402 tablet.cc:2369] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:29:04.547190 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.543665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.0.98.2:35439)
0926 07:29:04.543803 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:04.543809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:04.543826 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:29:04.543920 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:04.543923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:04.543977 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:04.544103 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:04.544347 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.544841 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.545702 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:04.545927 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:04.546035 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:04.546344 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:04.546739 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:04.546908 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:04.546969 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 07:29:04.547004 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:29:04.552803 19486 raft_consensus.cc:1261] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee [term 1 FOLLOWER]: Refusing update from remote peer bb4171cc7cc0400e8d0738aa3dd8bd4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:04.552801 19224 raft_consensus.cc:1261] T 8328255c8189466b915ac3e688104398 P 3dd2bbf29e40414ea7adcf99ed46c5b2 [term 1 FOLLOWER]: Refusing update from remote peer bb4171cc7cc0400e8d0738aa3dd8bd4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:04.553478 19557 consensus_queue.cc:1035] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd2bbf29e40414ea7adcf99ed46c5b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:04.553654 19559 consensus_queue.cc:1035] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbae1427159e4df983e07351d45034ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:04.559887 19562 mvcc.cc:205] Tried to move back new op lower bound from 7333510121681358848 to 7333510121466130432. Current Snapshot: MvccSnapshot[applied={T|T < 7333510121681358848}]
I20260926 07:29:04.560039 19564 mvcc.cc:205] Tried to move back new op lower bound from 7333510121681358848 to 7333510121466130432. Current Snapshot: MvccSnapshot[applied={T|T < 7333510121681358848}]
I20260926 07:29:04.564481 19563 mvcc.cc:205] Tried to move back new op lower bound from 7333510121681358848 to 7333510121466130432. Current Snapshot: MvccSnapshot[applied={T|T < 7333510121681358848}]
I20260926 07:29:04.568111 19081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:04.568270 19081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:04.570533 19081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2036b9d5505345c1b250d5f9c0dede45: Sending DeleteTablet for 3 replicas of tablet 8328255c8189466b915ac3e688104398
I20260926 07:29:04.571334 19335 tablet_service.cc:1514] Processing DeleteTablet for tablet 8328255c8189466b915ac3e688104398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:04 UTC) from {username='slave'} at 127.0.0.1:55208
I20260926 07:29:04.571421 19204 tablet_service.cc:1514] Processing DeleteTablet for tablet 8328255c8189466b915ac3e688104398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:04 UTC) from {username='slave'} at 127.0.0.1:41530
I20260926 07:29:04.571866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 19142
I20260926 07:29:04.571872 19466 tablet_service.cc:1514] Processing DeleteTablet for tablet 8328255c8189466b915ac3e688104398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:04 UTC) from {username='slave'} at 127.0.0.1:59712
I20260926 07:29:04.573330 19572 tablet_replica.cc:330] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e: stopping tablet replica
I20260926 07:29:04.573647 19572 raft_consensus.cc:2229] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:29:04.573998 19572 raft_consensus.cc:2258] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:04.575580 19572 ts_tablet_manager.cc:1905] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:04.579150 19572 ts_tablet_manager.cc:1918] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:04.579290 19572 log.cc:1196] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/8328255c8189466b915ac3e688104398
I20260926 07:29:04.579607 19572 ts_tablet_manager.cc:1939] T 8328255c8189466b915ac3e688104398 P bb4171cc7cc0400e8d0738aa3dd8bd4e: Deleting consensus metadata
W20260926 07:29:04.580147 19067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34431 (error 108)
I20260926 07:29:04.580956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 19273
I20260926 07:29:04.581219 19574 tablet_replica.cc:330] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee: stopping tablet replica
I20260926 07:29:04.581521 19574 raft_consensus.cc:2229] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:04.581852 19574 raft_consensus.cc:2258] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:04.582442 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.581256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34431 (local address 127.0.0.1:41544)
0926 07:29:04.582294 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:04.582364 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":934}
W20260926 07:29:04.582576 19067 catalog_manager.cc:4584] TS 3dd2bbf29e40414ea7adcf99ed46c5b2 (127.0.98.1:34431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8328255c8189466b915ac3e688104398: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34431: connect: Connection refused (error 111)
I20260926 07:29:04.583370 19574 ts_tablet_manager.cc:1905] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:04.585884 19574 ts_tablet_manager.cc:1918] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:04.585976 19574 log.cc:1196] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/8328255c8189466b915ac3e688104398
I20260926 07:29:04.586251 19574 ts_tablet_manager.cc:1939] T 8328255c8189466b915ac3e688104398 P dbae1427159e4df983e07351d45034ee: Deleting consensus metadata
I20260926 07:29:04.587420 19068 catalog_manager.cc:4841] TS dbae1427159e4df983e07351d45034ee (127.0.98.3:34487): tablet 8328255c8189466b915ac3e688104398 (table test-table [id=9dcf91072e19420686c3f5c30577e2db]) successfully deleted
I20260926 07:29:04.591684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 19404
I20260926 07:29:04.593572 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:04.591946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35439 (local address 127.0.0.1:55228)
0926 07:29:04.593448 (+ 1502us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:04.593509 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1397}
I20260926 07:29:04.597754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 19051
2026-09-26T07:29:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1223 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:04 dist-test-slave-g04g krb5kdc[19578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:04 dist-test-slave-g04g krb5kdc[19578](info): set up 2 sockets
Sep 26 07:29:04 dist-test-slave-g04g krb5kdc[19578](info): commencing operation
krb5kdc: starting...
W20260926 07:29:05.651083 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:05 dist-test-slave-g04g krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:29:05.771524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37927
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:05.881855 19591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:05.882175 19591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:05.882256 19591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:05.886332 19591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:05.886415 19591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:05.886454 19591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:05.886520 19591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:05.886579 19591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:05.891917 19591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:05.892963 19591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:05.893967 19591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:05.902866 19599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:05.902925 19597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:05.902949 19596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:05.903141 19591 server_base.cc:1034] running on GCE node
I20260926 07:29:05.903728 19591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:05.904791 19591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:05.905948 19591 hybrid_clock.cc:648] HybridClock initialized: now 1790407745905920 us; error 47 us; skew 500 ppm
Sep 26 07:29:05 dist-test-slave-g04g krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:05.908627 19591 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:29:05.909479 19591 webserver.cc:460] Webserver started at http://127.0.98.62:39079/ using document root <none> and password file <none>
I20260926 07:29:05.910020 19591 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:05.910106 19591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:05.910326 19591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:05.912367 19591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "896710a99d7f4837a2c719e84514b820"
format_stamp: "Formatted at 2026-09-26 07:29:05 on dist-test-slave-g04g"
server_key: "8055d474351fa47c39df22d0706be523"
server_key_iv: "bcc2d84e80f87662ab84ed9c4176e73d"
server_key_version: "encryptionkey@0"
I20260926 07:29:05.912911 19591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "896710a99d7f4837a2c719e84514b820"
format_stamp: "Formatted at 2026-09-26 07:29:05 on dist-test-slave-g04g"
server_key: "8055d474351fa47c39df22d0706be523"
server_key_iv: "bcc2d84e80f87662ab84ed9c4176e73d"
server_key_version: "encryptionkey@0"
I20260926 07:29:05.916697 19591 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 07:29:05.918926 19606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:05.920141 19591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:05.920277 19591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "896710a99d7f4837a2c719e84514b820"
format_stamp: "Formatted at 2026-09-26 07:29:05 on dist-test-slave-g04g"
server_key: "8055d474351fa47c39df22d0706be523"
server_key_iv: "bcc2d84e80f87662ab84ed9c4176e73d"
server_key_version: "encryptionkey@0"
I20260926 07:29:05.920387 19591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:05.949407 19591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:05.952559 19591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:05.952719 19591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:05.960971 19591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37927
I20260926 07:29:05.960973 19658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37927 every 8 connection(s)
I20260926 07:29:05.962343 19591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:05.965340 19659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:05.967644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 19591
I20260926 07:29:05.967784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:05.968082 392 external_mini_cluster.cc:1428] Setting key aa7ffe5e1f358e5613f508fa5a41cf09
I20260926 07:29:05.971683 19659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820: Bootstrap starting.
I20260926 07:29:05.974102 19659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:05.974927 19659 log.cc:824] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:29:05 dist-test-slave-g04g krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:05.977061 19659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820: No bootstrap required, opened a new log
I20260926 07:29:05.980738 19659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896710a99d7f4837a2c719e84514b820" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37927 } }
I20260926 07:29:05.980991 19659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:05.981060 19659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896710a99d7f4837a2c719e84514b820, State: Initialized, Role: FOLLOWER
I20260926 07:29:05.981513 19659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896710a99d7f4837a2c719e84514b820" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37927 } }
I20260926 07:29:05.981669 19659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:05.981747 19659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:05.981873 19659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:05.982003 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:05.969414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42516 (local address 127.0.98.62:37927)
0926 07:29:05.969804 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:05.969820 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:05.969854 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:29:05.970665 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:05.970683 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:05.971000 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:05.971343 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:05.972187 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:05.973141 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:05.974320 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:05.974789 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:05.974903 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:05.977395 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:05.977400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:05.977413 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:05.977438 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:05.979160 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:05.979589 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:05.979601 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:05.979606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:05.979710 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:05.979989 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:05.979994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:05.979996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:05.980376 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:05.980596 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:05.980819 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 07:29:05.981100 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":120,"threads_started":1}
I20260926 07:29:05.982790 19659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896710a99d7f4837a2c719e84514b820" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37927 } }
I20260926 07:29:05.983199 19659 leader_election.cc:302] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 896710a99d7f4837a2c719e84514b820; no voters:
I20260926 07:29:05.983569 19659 leader_election.cc:288] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:05.983683 19664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:05.983873 19664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [term 1 LEADER]: Becoming Leader. State: Replica: 896710a99d7f4837a2c719e84514b820, State: Running, Role: LEADER
I20260926 07:29:05.984184 19664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896710a99d7f4837a2c719e84514b820" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37927 } }
I20260926 07:29:05.984825 19659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:05.985801 19666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 896710a99d7f4837a2c719e84514b820. Latest consensus state: current_term: 1 leader_uuid: "896710a99d7f4837a2c719e84514b820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896710a99d7f4837a2c719e84514b820" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37927 } } }
I20260926 07:29:05.985945 19666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:05.986323 19673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:05.987433 19665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "896710a99d7f4837a2c719e84514b820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "896710a99d7f4837a2c719e84514b820" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37927 } } }
I20260926 07:29:05.987527 19665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:05.989575 19673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:05.994818 19673 catalog_manager.cc:1347] Generated new cluster ID: 549af3fa934d4414b10711e1ae14098f
I20260926 07:29:05.994884 19673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:06.001634 19673 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:06.002480 19673 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:06.010977 19673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820: Generated new TSK 0
I20260926 07:29:06.011643 19673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 07:29:06.042333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37927
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:06.153888 19685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:06.154150 19685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:06.154196 19685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:06.158314 19685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:06.158380 19685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:06.158480 19685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:29:06.164311 19685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:06.165338 19685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:06.166363 19685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:06.176152 19690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:06.176198 19691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:06.176194 19693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:06.176463 19685 server_base.cc:1034] running on GCE node
I20260926 07:29:06.176858 19685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:06.177454 19685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:06.178640 19685 hybrid_clock.cc:648] HybridClock initialized: now 1790407746178615 us; error 34 us; skew 500 ppm
Sep 26 07:29:06 dist-test-slave-g04g krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:06.181295 19685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 07:29:06.182080 19685 webserver.cc:460] Webserver started at http://127.0.98.1:35559/ using document root <none> and password file <none>
I20260926 07:29:06.182595 19685 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:06.182653 19685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:06.182813 19685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:06.184837 19685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "0b3aadfdeab64e96a348c5e1c9185bd7"
format_stamp: "Formatted at 2026-09-26 07:29:06 on dist-test-slave-g04g"
server_key: "29f4b66b6b9514cbbb3221f323ee26eb"
server_key_iv: "e86294fb981d3037d50059b3c7ce8312"
server_key_version: "encryptionkey@0"
I20260926 07:29:06.185437 19685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0b3aadfdeab64e96a348c5e1c9185bd7"
format_stamp: "Formatted at 2026-09-26 07:29:06 on dist-test-slave-g04g"
server_key: "29f4b66b6b9514cbbb3221f323ee26eb"
server_key_iv: "e86294fb981d3037d50059b3c7ce8312"
server_key_version: "encryptionkey@0"
I20260926 07:29:06.189267 19685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 07:29:06.191636 19700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:06.192828 19685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:06.192920 19685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "0b3aadfdeab64e96a348c5e1c9185bd7"
format_stamp: "Formatted at 2026-09-26 07:29:06 on dist-test-slave-g04g"
server_key: "29f4b66b6b9514cbbb3221f323ee26eb"
server_key_iv: "e86294fb981d3037d50059b3c7ce8312"
server_key_version: "encryptionkey@0"
I20260926 07:29:06.193046 19685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:06.219576 19685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:06.222841 19685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:06.222999 19685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:06.223620 19685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:06.224478 19685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:06.224525 19685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:06.224598 19685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:06.224630 19685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:06.233587 19685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41101
I20260926 07:29:06.233608 19813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41101 every 8 connection(s)
I20260926 07:29:06.234509 19685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:29:06.238624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 19685
I20260926 07:29:06.238768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:29:06.239156 392 external_mini_cluster.cc:1428] Setting key 03de9c4141bf3ee191180bd909c40cc1
Sep 26 07:29:06 dist-test-slave-g04g krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:06.249490 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.236326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47127 (local address 127.0.98.62:37927)
0926 07:29:06.236480 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:06.236485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:06.237376 (+ 891us) server_negotiation.cc:407] Connection header received
0926 07:29:06.238667 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:06.238674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:06.238740 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:06.238837 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:06.240121 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.240823 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.242590 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.242846 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.242959 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.245607 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:06.245613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:06.245620 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:06.245653 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:06.247427 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:06.247911 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:06.247917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:06.247920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:06.248007 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:06.248354 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:06.248358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:06.248359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:06.248583 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:06.248699 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:06.249168 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 07:29:06.249320 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 07:29:06.250249 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.236784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37927 (local address 127.0.98.1:47127)
0926 07:29:06.237223 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:06.237266 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:06.238414 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:06.239018 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:06.239066 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:06.239494 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:06.239989 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.240002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.240952 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.240959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.242466 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.242475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.242994 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.242998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.243213 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.243756 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:06.243782 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:06.245440 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:06.247542 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:06.247546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:06.247559 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:06.247804 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:06.248105 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:06.248108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:06.248109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:06.248257 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:06.248693 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:06.248698 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:06.248923 (+ 225us) client_negotiation.cc:769] Sending connection context
0926 07:29:06.249087 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 07:29:06.249363 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
I20260926 07:29:06.251768 19814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37927
I20260926 07:29:06.252018 19814 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 07:29:06.252604 19814 heartbeater.cc:511] Master 127.0.98.62:37927 requested a full tablet report, sending...
I20260926 07:29:06.254138 19623 ts_manager.cc:194] Registered new tserver with Master: 0b3aadfdeab64e96a348c5e1c9185bd7 (127.0.98.1:41101)
I20260926 07:29:06.255362 19623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 07:29:06.270530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37927
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:06.382920 19819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:06.383256 19819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:06.383350 19819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:06.387676 19819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:06.387766 19819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:06.387854 19819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:29:06.393316 19819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:06.394435 19819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:06.395524 19819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:06.405479 19824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:06.405592 19819 server_base.cc:1034] running on GCE node
W20260926 07:29:06.405473 19825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:06.405478 19827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:06.406111 19819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:06.406796 19819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:06.407956 19819 hybrid_clock.cc:648] HybridClock initialized: now 1790407746407932 us; error 37 us; skew 500 ppm
Sep 26 07:29:06 dist-test-slave-g04g krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:06.410545 19819 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 07:29:06.411491 19819 webserver.cc:460] Webserver started at http://127.0.98.2:44313/ using document root <none> and password file <none>
I20260926 07:29:06.412046 19819 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:06.412132 19819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:06.412359 19819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:06.414350 19819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "3d26e454d60c446bb17e8cb4cf55318e"
format_stamp: "Formatted at 2026-09-26 07:29:06 on dist-test-slave-g04g"
server_key: "76c3833b786df4718ef8ba1e3bf4d053"
server_key_iv: "5ca4f14fd222940f2569bdb42d5e4c5b"
server_key_version: "encryptionkey@0"
I20260926 07:29:06.414909 19819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3d26e454d60c446bb17e8cb4cf55318e"
format_stamp: "Formatted at 2026-09-26 07:29:06 on dist-test-slave-g04g"
server_key: "76c3833b786df4718ef8ba1e3bf4d053"
server_key_iv: "5ca4f14fd222940f2569bdb42d5e4c5b"
server_key_version: "encryptionkey@0"
I20260926 07:29:06.418840 19819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 07:29:06.421244 19834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:06.422513 19819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:06.422645 19819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "3d26e454d60c446bb17e8cb4cf55318e"
format_stamp: "Formatted at 2026-09-26 07:29:06 on dist-test-slave-g04g"
server_key: "76c3833b786df4718ef8ba1e3bf4d053"
server_key_iv: "5ca4f14fd222940f2569bdb42d5e4c5b"
server_key_version: "encryptionkey@0"
I20260926 07:29:06.422755 19819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:06.439881 19819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:06.442728 19819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:06.442934 19819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:06.443549 19819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:06.444377 19819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:06.444446 19819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:06.444516 19819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:06.444567 19819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:06.453755 19819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44523
I20260926 07:29:06.453778 19947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44523 every 8 connection(s)
I20260926 07:29:06.454691 19819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:29:06.457302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 19819
I20260926 07:29:06.457432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:29:06.457747 392 external_mini_cluster.cc:1428] Setting key 5ce9a9115247de5ba4d2903411defa79
Sep 26 07:29:06 dist-test-slave-g04g krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 07:29:06.469060 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.456555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33425 (local address 127.0.98.62:37927)
0926 07:29:06.456693 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:06.456696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:06.457586 (+ 890us) server_negotiation.cc:407] Connection header received
0926 07:29:06.459077 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:06.459080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:06.459136 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:06.459220 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:06.460804 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.461326 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.462523 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.462750 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.462872 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.465393 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:06.465396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:06.465398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:06.465423 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:06.466877 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:06.467478 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:06.467482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:06.467484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:06.467582 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:06.467951 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:06.467954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:06.467956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:06.468170 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:06.468294 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:06.468828 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 07:29:06.468935 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:29:06.469978 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.456889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37927 (local address 127.0.98.2:33425)
0926 07:29:06.457398 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:06.457457 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:06.458842 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:06.459394 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:06.459401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:06.459910 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:06.460629 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.460652 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.461467 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.461472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.462391 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.462400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.462907 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.462912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.463137 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.463751 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:06.463769 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:06.465202 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:06.466998 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:06.467004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:06.467019 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:06.467388 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:06.467681 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:06.467685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:06.467687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:06.467844 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:06.468314 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:06.468320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:06.468570 (+ 250us) client_negotiation.cc:769] Sending connection context
0926 07:29:06.468754 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 07:29:06.469023 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":163,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 07:29:06.471359 19948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37927
I20260926 07:29:06.471635 19948 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:06.472261 19948 heartbeater.cc:511] Master 127.0.98.62:37927 requested a full tablet report, sending...
I20260926 07:29:06.473394 19623 ts_manager.cc:194] Registered new tserver with Master: 3d26e454d60c446bb17e8cb4cf55318e (127.0.98.2:44523)
I20260926 07:29:06.473964 19623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 07:29:06.486716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37927
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:06.610337 19953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:06.610592 19953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:06.610637 19953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:06.615204 19953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:06.615262 19953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:06.615324 19953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:29:06.622553 19953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:06.624039 19953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:06.625372 19953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:06.636268 19961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:06.636512 19958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:06.636508 19959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:06.636488 19953 server_base.cc:1034] running on GCE node
I20260926 07:29:06.636914 19953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:06.637535 19953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:06.638715 19953 hybrid_clock.cc:648] HybridClock initialized: now 1790407746638703 us; error 23 us; skew 500 ppm
Sep 26 07:29:06 dist-test-slave-g04g krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:06.641497 19953 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 07:29:06.642362 19953 webserver.cc:460] Webserver started at http://127.0.98.3:42393/ using document root <none> and password file <none>
I20260926 07:29:06.642904 19953 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:06.642992 19953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:06.643240 19953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:06.645218 19953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "d32b8deabdb1488fb8662acabde1e749"
format_stamp: "Formatted at 2026-09-26 07:29:06 on dist-test-slave-g04g"
server_key: "11f5ace6923880ac355ea4b28e26dcc6"
server_key_iv: "e6955252353585701458875eeb377d99"
server_key_version: "encryptionkey@0"
I20260926 07:29:06.645807 19953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d32b8deabdb1488fb8662acabde1e749"
format_stamp: "Formatted at 2026-09-26 07:29:06 on dist-test-slave-g04g"
server_key: "11f5ace6923880ac355ea4b28e26dcc6"
server_key_iv: "e6955252353585701458875eeb377d99"
server_key_version: "encryptionkey@0"
I20260926 07:29:06.649435 19953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:29:06.651787 19968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:06.652957 19953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:06.653085 19953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "d32b8deabdb1488fb8662acabde1e749"
format_stamp: "Formatted at 2026-09-26 07:29:06 on dist-test-slave-g04g"
server_key: "11f5ace6923880ac355ea4b28e26dcc6"
server_key_iv: "e6955252353585701458875eeb377d99"
server_key_version: "encryptionkey@0"
I20260926 07:29:06.653191 19953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:06.667752 19953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:06.670969 19953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:06.671185 19953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:06.671761 19953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:06.672556 19953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:06.672626 19953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:06.672693 19953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:06.672740 19953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:06.681420 19953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38947
I20260926 07:29:06.681445 20081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38947 every 8 connection(s)
I20260926 07:29:06.682284 19953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:29:06.683593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 19953
I20260926 07:29:06.683735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:29:06.684038 392 external_mini_cluster.cc:1428] Setting key 3bdf86ccb812aa861f748e98a40cf6ec
Sep 26 07:29:06 dist-test-slave-g04g krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:06.695389 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.684227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58427 (local address 127.0.98.62:37927)
0926 07:29:06.684380 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:06.684386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:06.685279 (+ 893us) server_negotiation.cc:407] Connection header received
0926 07:29:06.686702 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:06.686704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:06.686748 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:06.686816 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:06.688067 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.688589 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.689661 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.689875 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.689960 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.692245 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:06.692248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:06.692250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:06.692273 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:06.693701 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:06.694127 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:06.694129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:06.694130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:06.694198 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:06.694477 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:06.694479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:06.694481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:06.694638 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:06.694711 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:06.695186 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 07:29:06.695274 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:29:06.696184 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.684660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37927 (local address 127.0.98.3:58427)
0926 07:29:06.685143 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:06.685194 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:06.686524 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:06.687108 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:06.687116 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:06.687483 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:06.687946 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.687958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.688691 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.688693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.689555 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.689563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.690002 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.690005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.690136 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.690640 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:06.690655 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:06.692106 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:06.693793 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:06.693796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:06.693806 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:06.694046 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:06.694272 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:06.694274 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:06.694276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:06.694397 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:06.694719 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:06.694724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:06.694942 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 07:29:06.695153 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 07:29:06.695402 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":138,"threads_started":1}
I20260926 07:29:06.698105 20082 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37927
I20260926 07:29:06.698365 20082 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:06.698858 20082 heartbeater.cc:511] Master 127.0.98.62:37927 requested a full tablet report, sending...
I20260926 07:29:06.699826 19622 ts_manager.cc:194] Registered new tserver with Master: d32b8deabdb1488fb8662acabde1e749 (127.0.98.3:38947)
I20260926 07:29:06.700367 19622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58427
I20260926 07:29:06.710232 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:29:06.719384 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.712336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.0.98.62:37927)
0926 07:29:06.712489 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:06.712494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:06.712626 (+ 132us) server_negotiation.cc:407] Connection header received
0926 07:29:06.712721 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:06.712723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:06.712767 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:06.712831 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:06.713493 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.713976 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.714750 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.714951 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.715022 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.716349 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:06.716354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:06.716357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:06.716392 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:06.717928 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:06.718308 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:06.718312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:06.718314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:06.718391 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:06.718635 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:06.718637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:06.718638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:06.718842 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:06.718945 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:06.719165 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 07:29:06.719256 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:29:06.721974 19622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:29:06.724830 19622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:06.738529 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.733143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40400 (local address 127.0.98.3:38947)
0926 07:29:06.733932 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:06.733945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:06.733974 (+ 29us) server_negotiation.cc:407] Connection header received
0926 07:29:06.734054 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:06.734061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:06.734269 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:06.734429 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:06.734724 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.735905 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.737274 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.737874 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.737989 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.738127 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:06.738285 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 07:29:06.738377 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":112,"threads_started":1}
I20260926 07:29:06.739135 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.733009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38947 (local address 127.0.0.1:40400)
0926 07:29:06.733739 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:06.733786 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:06.733911 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:06.734442 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:06.734449 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:06.734475 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:06.734612 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.734618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.736052 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.736056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.737152 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.737161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.737995 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.737999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.738100 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.738138 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 07:29:06.738246 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 07:29:06.738352 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":4,"thread_start_us":67,"threads_started":1}
I20260926 07:29:06.739364 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.733524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41101 (local address 127.0.0.1:53238)
0926 07:29:06.734248 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:06.734261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:06.734347 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:06.734879 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:06.734882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:06.734898 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:06.735013 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.735050 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.736352 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.736355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.737647 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.737657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.738897 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.738901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.739006 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.739085 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 07:29:06.739157 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 07:29:06.739236 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":46,"thread_start_us":82,"threads_started":1}
I20260926 07:29:06.739703 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.733196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44523 (local address 127.0.0.1:47302)
0926 07:29:06.734054 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:06.734067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:06.734426 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:06.735082 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:06.735088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:06.735110 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:06.735209 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.735215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.736586 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.736590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.737772 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.737780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.739022 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.739025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.739178 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.739205 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 07:29:06.739299 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 07:29:06.739367 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":47,"thread_start_us":90,"threads_started":1}
I20260926 07:29:06.739719 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.733750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.0.98.2:44523)
0926 07:29:06.734220 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:06.734231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:06.734254 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:29:06.734588 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:06.734597 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:06.734818 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:06.734963 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:06.735323 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.736454 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.737882 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.738714 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.738811 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.738960 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:06.739462 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 07:29:06.739559 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":135,"threads_started":1}
I20260926 07:29:06.739827 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.733699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53238 (local address 127.0.98.1:41101)
0926 07:29:06.734304 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:06.734325 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:06.734348 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:29:06.734532 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:06.734538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:06.734734 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:06.734900 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:06.735179 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.736204 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.737778 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.738609 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.738715 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.738837 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:06.739590 (+ 753us) server_negotiation.cc:299] Negotiation successful
0926 07:29:06.739690 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":114,"threads_started":1}
I20260926 07:29:06.742034 19748 tablet_service.cc:1467] Processing CreateTablet for tablet ecaa89c46d1f495a9e95376c7e0a43ca (DEFAULT_TABLE table=test-table [id=864197be8f224334a5158bea3a156d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:06.742031 20016 tablet_service.cc:1467] Processing CreateTablet for tablet ecaa89c46d1f495a9e95376c7e0a43ca (DEFAULT_TABLE table=test-table [id=864197be8f224334a5158bea3a156d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:06.743115 20016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecaa89c46d1f495a9e95376c7e0a43ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:06.743129 19748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecaa89c46d1f495a9e95376c7e0a43ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:06.744346 19882 tablet_service.cc:1467] Processing CreateTablet for tablet ecaa89c46d1f495a9e95376c7e0a43ca (DEFAULT_TABLE table=test-table [id=864197be8f224334a5158bea3a156d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:06.745337 19882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecaa89c46d1f495a9e95376c7e0a43ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:06.749311 20099 tablet_bootstrap.cc:492] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7: Bootstrap starting.
I20260926 07:29:06.749948 20100 tablet_bootstrap.cc:492] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749: Bootstrap starting.
I20260926 07:29:06.751924 20101 tablet_bootstrap.cc:492] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e: Bootstrap starting.
I20260926 07:29:06.752408 20100 tablet_bootstrap.cc:654] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:06.752415 20099 tablet_bootstrap.cc:654] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:06.753387 20099 log.cc:824] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:06.753664 20100 log.cc:824] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:06.753831 20101 tablet_bootstrap.cc:654] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:06.754890 20101 log.cc:824] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:06.755448 20100 tablet_bootstrap.cc:492] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749: No bootstrap required, opened a new log
I20260926 07:29:06.755448 20099 tablet_bootstrap.cc:492] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7: No bootstrap required, opened a new log
I20260926 07:29:06.755640 20099 ts_tablet_manager.cc:1397] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260926 07:29:06.755640 20100 ts_tablet_manager.cc:1397] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:29:06.756747 20101 tablet_bootstrap.cc:492] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e: No bootstrap required, opened a new log
I20260926 07:29:06.756919 20101 ts_tablet_manager.cc:1397] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 07:29:06.759678 20100 raft_consensus.cc:348] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32b8deabdb1488fb8662acabde1e749" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } } peers { permanent_uuid: "3d26e454d60c446bb17e8cb4cf55318e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44523 } } peers { permanent_uuid: "0b3aadfdeab64e96a348c5e1c9185bd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } }
I20260926 07:29:06.759778 20101 raft_consensus.cc:348] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32b8deabdb1488fb8662acabde1e749" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } } peers { permanent_uuid: "3d26e454d60c446bb17e8cb4cf55318e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44523 } } peers { permanent_uuid: "0b3aadfdeab64e96a348c5e1c9185bd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } }
I20260926 07:29:06.759971 20100 raft_consensus.cc:374] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:06.760004 20101 raft_consensus.cc:374] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:06.760025 20100 raft_consensus.cc:729] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32b8deabdb1488fb8662acabde1e749, State: Initialized, Role: FOLLOWER
I20260926 07:29:06.760051 20101 raft_consensus.cc:729] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d26e454d60c446bb17e8cb4cf55318e, State: Initialized, Role: FOLLOWER
I20260926 07:29:06.759940 20099 raft_consensus.cc:348] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32b8deabdb1488fb8662acabde1e749" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } } peers { permanent_uuid: "3d26e454d60c446bb17e8cb4cf55318e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44523 } } peers { permanent_uuid: "0b3aadfdeab64e96a348c5e1c9185bd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } }
I20260926 07:29:06.760156 20099 raft_consensus.cc:374] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:06.760210 20099 raft_consensus.cc:729] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3aadfdeab64e96a348c5e1c9185bd7, State: Initialized, Role: FOLLOWER
I20260926 07:29:06.760469 20100 consensus_queue.cc:260] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32b8deabdb1488fb8662acabde1e749" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } } peers { permanent_uuid: "3d26e454d60c446bb17e8cb4cf55318e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44523 } } peers { permanent_uuid: "0b3aadfdeab64e96a348c5e1c9185bd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } }
I20260926 07:29:06.760497 20101 consensus_queue.cc:260] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32b8deabdb1488fb8662acabde1e749" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } } peers { permanent_uuid: "3d26e454d60c446bb17e8cb4cf55318e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44523 } } peers { permanent_uuid: "0b3aadfdeab64e96a348c5e1c9185bd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } }
I20260926 07:29:06.760563 20099 consensus_queue.cc:260] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32b8deabdb1488fb8662acabde1e749" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } } peers { permanent_uuid: "3d26e454d60c446bb17e8cb4cf55318e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44523 } } peers { permanent_uuid: "0b3aadfdeab64e96a348c5e1c9185bd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } }
I20260926 07:29:06.761122 20082 heartbeater.cc:503] Master 127.0.98.62:37927 was elected leader, sending a full tablet report...
I20260926 07:29:06.761504 20100 ts_tablet_manager.cc:1428] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:29:06.761504 20099 ts_tablet_manager.cc:1428] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:29:06.761840 19814 heartbeater.cc:503] Master 127.0.98.62:37927 was elected leader, sending a full tablet report...
I20260926 07:29:06.762195 19948 heartbeater.cc:503] Master 127.0.98.62:37927 was elected leader, sending a full tablet report...
I20260926 07:29:06.762286 20101 ts_tablet_manager.cc:1428] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 07:29:06.933249 20083 tablet.cc:2369] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:29:06.943511 20105 raft_consensus.cc:482] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:06.943763 20105 raft_consensus.cc:504] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32b8deabdb1488fb8662acabde1e749" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } } peers { permanent_uuid: "3d26e454d60c446bb17e8cb4cf55318e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44523 } } peers { permanent_uuid: "0b3aadfdeab64e96a348c5e1c9185bd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } }
I20260926 07:29:06.944888 20105 leader_election.cc:288] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d26e454d60c446bb17e8cb4cf55318e (127.0.98.2:44523), 0b3aadfdeab64e96a348c5e1c9185bd7 (127.0.98.1:41101)
I20260926 07:29:06.949182 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.945181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44523 (local address 127.0.98.3:45467)
0926 07:29:06.945598 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:06.945609 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:06.945734 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:06.945982 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:06.945985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:06.946005 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:06.946127 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.946133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.946812 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.946818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.947662 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.947670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.948504 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.948507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.948666 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.948842 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 07:29:06.948925 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 07:29:06.948998 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":152,"thread_start_us":139,"threads_started":1}
I20260926 07:29:06.949263 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.945293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45467 (local address 127.0.98.2:44523)
0926 07:29:06.945451 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:06.945455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:06.945692 (+ 237us) server_negotiation.cc:407] Connection header received
0926 07:29:06.945839 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:06.945842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:06.945890 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:06.945974 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:06.946229 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.946707 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.947784 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.948380 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.948486 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.948556 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:06.948988 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 07:29:06.949118 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 07:29:06.949415 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.945288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60591 (local address 127.0.98.1:41101)
0926 07:29:06.945445 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:06.945449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:06.945466 (+ 17us) server_negotiation.cc:407] Connection header received
0926 07:29:06.945635 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:06.945638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:06.945680 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:06.945765 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:06.946038 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.946544 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.947642 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.948174 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.948274 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.948342 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:06.949224 (+ 882us) server_negotiation.cc:299] Negotiation successful
0926 07:29:06.949303 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:29:06.949617 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.945181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41101 (local address 127.0.98.3:60591)
0926 07:29:06.945382 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:06.945398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:06.945497 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:06.945774 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:06.945777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:06.945794 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:06.945914 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.945919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.946652 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.946657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.947521 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:06.947529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.948428 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.948431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:06.948699 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.948831 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 07:29:06.949037 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 07:29:06.949110 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":86}
I20260926 07:29:06.950330 19902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa89c46d1f495a9e95376c7e0a43ca" candidate_uuid: "d32b8deabdb1488fb8662acabde1e749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d26e454d60c446bb17e8cb4cf55318e" is_pre_election: true
I20260926 07:29:06.950455 19768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa89c46d1f495a9e95376c7e0a43ca" candidate_uuid: "d32b8deabdb1488fb8662acabde1e749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b3aadfdeab64e96a348c5e1c9185bd7" is_pre_election: true
I20260926 07:29:06.950655 19902 raft_consensus.cc:2454] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d32b8deabdb1488fb8662acabde1e749 in term 0.
I20260926 07:29:06.950752 19768 raft_consensus.cc:2454] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d32b8deabdb1488fb8662acabde1e749 in term 0.
I20260926 07:29:06.951190 19970 leader_election.cc:302] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d26e454d60c446bb17e8cb4cf55318e, d32b8deabdb1488fb8662acabde1e749; no voters:
I20260926 07:29:06.951454 20105 raft_consensus.cc:2781] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:06.951543 20105 raft_consensus.cc:482] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:06.951597 20105 raft_consensus.cc:3037] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:06.952566 20105 raft_consensus.cc:504] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32b8deabdb1488fb8662acabde1e749" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } } peers { permanent_uuid: "3d26e454d60c446bb17e8cb4cf55318e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44523 } } peers { permanent_uuid: "0b3aadfdeab64e96a348c5e1c9185bd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } }
I20260926 07:29:06.952891 20105 leader_election.cc:288] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [CANDIDATE]: Term 1 election: Requested vote from peers 3d26e454d60c446bb17e8cb4cf55318e (127.0.98.2:44523), 0b3aadfdeab64e96a348c5e1c9185bd7 (127.0.98.1:41101)
I20260926 07:29:06.953184 19902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa89c46d1f495a9e95376c7e0a43ca" candidate_uuid: "d32b8deabdb1488fb8662acabde1e749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d26e454d60c446bb17e8cb4cf55318e"
I20260926 07:29:06.953253 19768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa89c46d1f495a9e95376c7e0a43ca" candidate_uuid: "d32b8deabdb1488fb8662acabde1e749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b3aadfdeab64e96a348c5e1c9185bd7"
I20260926 07:29:06.953298 19902 raft_consensus.cc:3037] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:06.953392 19768 raft_consensus.cc:3037] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:06.954312 19902 raft_consensus.cc:2454] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d32b8deabdb1488fb8662acabde1e749 in term 1.
I20260926 07:29:06.954473 19768 raft_consensus.cc:2454] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d32b8deabdb1488fb8662acabde1e749 in term 1.
I20260926 07:29:06.954627 19970 leader_election.cc:302] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d26e454d60c446bb17e8cb4cf55318e, d32b8deabdb1488fb8662acabde1e749; no voters:
I20260926 07:29:06.954788 20105 raft_consensus.cc:2781] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:06.955008 20105 raft_consensus.cc:686] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [term 1 LEADER]: Becoming Leader. State: Replica: d32b8deabdb1488fb8662acabde1e749, State: Running, Role: LEADER
I20260926 07:29:06.955370 20105 consensus_queue.cc:237] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32b8deabdb1488fb8662acabde1e749" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } } peers { permanent_uuid: "3d26e454d60c446bb17e8cb4cf55318e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44523 } } peers { permanent_uuid: "0b3aadfdeab64e96a348c5e1c9185bd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } }
W20260926 07:29:06.955861 19949 tablet.cc:2369] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:29:06.958730 19623 catalog_manager.cc:5495] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 reported cstate change: term changed from 0 to 1, leader changed from <none> to d32b8deabdb1488fb8662acabde1e749 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d32b8deabdb1488fb8662acabde1e749" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d32b8deabdb1488fb8662acabde1e749" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d26e454d60c446bb17e8cb4cf55318e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b3aadfdeab64e96a348c5e1c9185bd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } health_report { overall_health: UNKNOWN } } }
I20260926 07:29:06.974093 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:06.970268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.0.98.3:38947)
0926 07:29:06.970415 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:06.970420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:06.970446 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:29:06.970618 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:06.970621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:06.970679 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:06.970782 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:06.970994 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.971786 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.972597 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:06.972932 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:06.973026 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:06.973294 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:06.973660 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:06.973781 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:06.973877 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 07:29:06.973930 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:29:06.979772 19902 raft_consensus.cc:1261] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e [term 1 FOLLOWER]: Refusing update from remote peer d32b8deabdb1488fb8662acabde1e749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:06.979799 19768 raft_consensus.cc:1261] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7 [term 1 FOLLOWER]: Refusing update from remote peer d32b8deabdb1488fb8662acabde1e749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:06.980405 20109 consensus_queue.cc:1035] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d26e454d60c446bb17e8cb4cf55318e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:06.980567 20105 consensus_queue.cc:1035] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b3aadfdeab64e96a348c5e1c9185bd7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:29:06.985788 19815 tablet.cc:2369] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:29:06.989140 20112 mvcc.cc:205] Tried to move back new op lower bound from 7333510131621404672 to 7333510131531567104. Current Snapshot: MvccSnapshot[applied={T|T < 7333510131621404672}]
I20260926 07:29:06.997288 19623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:06.997452 19623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:06.999795 19623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 896710a99d7f4837a2c719e84514b820: Sending DeleteTablet for 3 replicas of tablet ecaa89c46d1f495a9e95376c7e0a43ca
I20260926 07:29:07.000478 20016 tablet_service.cc:1514] Processing DeleteTablet for tablet ecaa89c46d1f495a9e95376c7e0a43ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40400
I20260926 07:29:07.000487 19882 tablet_service.cc:1514] Processing DeleteTablet for tablet ecaa89c46d1f495a9e95376c7e0a43ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47302
I20260926 07:29:07.000653 19748 tablet_service.cc:1514] Processing DeleteTablet for tablet ecaa89c46d1f495a9e95376c7e0a43ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53238
I20260926 07:29:07.001056 20123 tablet_replica.cc:330] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e: stopping tablet replica
I20260926 07:29:07.001180 20124 tablet_replica.cc:330] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7: stopping tablet replica
I20260926 07:29:07.001355 20123 raft_consensus.cc:2229] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:07.001572 20124 raft_consensus.cc:2229] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:07.001667 20123 raft_consensus.cc:2258] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:07.001765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 19685
I20260926 07:29:07.001986 20124 raft_consensus.cc:2258] T ecaa89c46d1f495a9e95376c7e0a43ca P 0b3aadfdeab64e96a348c5e1c9185bd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:07.003418 20123 ts_tablet_manager.cc:1905] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:07.006103 20123 ts_tablet_manager.cc:1918] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:07.006199 20123 log.cc:1196] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/ecaa89c46d1f495a9e95376c7e0a43ca
I20260926 07:29:07.006488 20123 ts_tablet_manager.cc:1939] T ecaa89c46d1f495a9e95376c7e0a43ca P 3d26e454d60c446bb17e8cb4cf55318e: Deleting consensus metadata
I20260926 07:29:07.006978 20122 tablet_replica.cc:330] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749: stopping tablet replica
I20260926 07:29:07.007333 20122 raft_consensus.cc:2229] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:29:07.007699 19608 catalog_manager.cc:4841] TS 3d26e454d60c446bb17e8cb4cf55318e (127.0.98.2:44523): tablet ecaa89c46d1f495a9e95376c7e0a43ca (table test-table [id=864197be8f224334a5158bea3a156d1d]) successfully deleted
I20260926 07:29:07.008155 20122 raft_consensus.cc:2258] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:07.009594 20122 ts_tablet_manager.cc:1905] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:29:07.010335 19609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41101 (error 108)
I20260926 07:29:07.011332 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.011004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41101 (local address 127.0.0.1:53252)
0926 07:29:07.011179 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:07.011248 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 07:29:07.011408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 19819
I20260926 07:29:07.014873 20122 ts_tablet_manager.cc:1918] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:07.014977 20122 log.cc:1196] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/ecaa89c46d1f495a9e95376c7e0a43ca
W20260926 07:29:07.015149 19609 catalog_manager.cc:4584] TS 0b3aadfdeab64e96a348c5e1c9185bd7 (127.0.98.1:41101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ecaa89c46d1f495a9e95376c7e0a43ca: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41101: connect: Connection refused (error 111)
I20260926 07:29:07.015336 20122 ts_tablet_manager.cc:1939] T ecaa89c46d1f495a9e95376c7e0a43ca P d32b8deabdb1488fb8662acabde1e749: Deleting consensus metadata
I20260926 07:29:07.016486 19607 catalog_manager.cc:4841] TS d32b8deabdb1488fb8662acabde1e749 (127.0.98.3:38947): tablet ecaa89c46d1f495a9e95376c7e0a43ca (table test-table [id=864197be8f224334a5158bea3a156d1d]) successfully deleted
I20260926 07:29:07.018247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 19953
I20260926 07:29:07.023671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 19591
2026-09-26T07:29:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2426 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T07:29:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:07Z Disabled control of system clock
I20260926 07:29:07.053725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39725
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:29:07.162333 20129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:07.162596 20129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:07.162634 20129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:07.166561 20129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:07.166615 20129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:07.166633 20129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:07.166652 20129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:07.166671 20129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:07.171830 20129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:07.172794 20129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:07.174077 20129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:07.182372 20135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:07.182385 20137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:07.182564 20129 server_base.cc:1034] running on GCE node
W20260926 07:29:07.182447 20134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:07.183275 20129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:07.184391 20129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:07.185567 20129 hybrid_clock.cc:648] HybridClock initialized: now 1790407747185533 us; error 47 us; skew 500 ppm
I20260926 07:29:07.187467 20129 webserver.cc:460] Webserver started at http://127.0.98.62:38521/ using document root <none> and password file <none>
I20260926 07:29:07.188037 20129 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:07.188130 20129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:07.188373 20129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:07.190382 20129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "803fe952d8014f118c7dd7d3da8099e6"
format_stamp: "Formatted at 2026-09-26 07:29:07 on dist-test-slave-g04g"
server_key: "caa4308e9704c02a847f581011208527"
server_key_iv: "f7216bda4f2748a038555e47a946c11a"
server_key_version: "encryptionkey@0"
I20260926 07:29:07.190948 20129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "803fe952d8014f118c7dd7d3da8099e6"
format_stamp: "Formatted at 2026-09-26 07:29:07 on dist-test-slave-g04g"
server_key: "caa4308e9704c02a847f581011208527"
server_key_iv: "f7216bda4f2748a038555e47a946c11a"
server_key_version: "encryptionkey@0"
I20260926 07:29:07.194429 20129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:29:07.196796 20143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:07.197945 20129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 07:29:07.198086 20129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "803fe952d8014f118c7dd7d3da8099e6"
format_stamp: "Formatted at 2026-09-26 07:29:07 on dist-test-slave-g04g"
server_key: "caa4308e9704c02a847f581011208527"
server_key_iv: "f7216bda4f2748a038555e47a946c11a"
server_key_version: "encryptionkey@0"
I20260926 07:29:07.198192 20129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:07.213943 20129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:07.214627 20129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:07.214833 20129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:07.221925 20129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39725
I20260926 07:29:07.221932 20195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39725 every 8 connection(s)
I20260926 07:29:07.222851 20129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:07.225922 20196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:07.229030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 20129
I20260926 07:29:07.229154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:07.229442 392 external_mini_cluster.cc:1428] Setting key e08e1aa4bd2eea00ae55723a3b0aaf0d
I20260926 07:29:07.232162 20196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6: Bootstrap starting.
I20260926 07:29:07.234938 20196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:07.235872 20196 log.cc:824] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:07.238236 20196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6: No bootstrap required, opened a new log
I20260926 07:29:07.239459 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.230878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.0.98.62:39725)
0926 07:29:07.231607 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:07.231624 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:07.231660 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:29:07.232343 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:07.232360 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:07.232795 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:07.233073 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:07.233418 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.234528 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.235898 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.236578 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.236756 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.237604 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:07.237609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:07.237622 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:07.238019 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:07.238078 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:07.238098 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:07.238181 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:07.238381 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 07:29:07.238679 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":184,"threads_started":1}
I20260926 07:29:07.242213 20196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803fe952d8014f118c7dd7d3da8099e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39725 } }
I20260926 07:29:07.242420 20196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:07.242494 20196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 803fe952d8014f118c7dd7d3da8099e6, State: Initialized, Role: FOLLOWER
I20260926 07:29:07.242913 20196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803fe952d8014f118c7dd7d3da8099e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39725 } }
I20260926 07:29:07.243073 20196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:07.243151 20196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:07.243287 20196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:07.244194 20196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803fe952d8014f118c7dd7d3da8099e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39725 } }
I20260926 07:29:07.244525 20196 leader_election.cc:302] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 803fe952d8014f118c7dd7d3da8099e6; no voters:
I20260926 07:29:07.244824 20196 leader_election.cc:288] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:07.244959 20201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:07.245188 20201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [term 1 LEADER]: Becoming Leader. State: Replica: 803fe952d8014f118c7dd7d3da8099e6, State: Running, Role: LEADER
I20260926 07:29:07.245498 20201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803fe952d8014f118c7dd7d3da8099e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39725 } }
I20260926 07:29:07.246102 20196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:07.247300 20203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 803fe952d8014f118c7dd7d3da8099e6. Latest consensus state: current_term: 1 leader_uuid: "803fe952d8014f118c7dd7d3da8099e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803fe952d8014f118c7dd7d3da8099e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39725 } } }
I20260926 07:29:07.247319 20202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "803fe952d8014f118c7dd7d3da8099e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803fe952d8014f118c7dd7d3da8099e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39725 } } }
I20260926 07:29:07.247449 20203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:07.247453 20202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:07.247848 20210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:07.250401 20210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:07.255807 20210 catalog_manager.cc:1347] Generated new cluster ID: 243dc6ba1ded4a078838c93d0c30c5ea
I20260926 07:29:07.255872 20210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:07.271456 20210 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:07.272428 20210 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:07.280777 20210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6: Generated new TSK 0
I20260926 07:29:07.281605 20210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:07.292924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39725
--builtin_ntp_servers=127.0.98.20:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:29:07.403930 20220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:07.404176 20220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:07.404213 20220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:07.408161 20220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:07.408205 20220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:07.408263 20220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 07:29:07.412840 20220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:07.413805 20220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:07.415128 20220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:07.424831 20225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:07.424798 20226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:07.424785 20228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:07.424976 20220 server_base.cc:1034] running on GCE node
I20260926 07:29:07.425468 20220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:07.426079 20220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:07.427263 20220 hybrid_clock.cc:648] HybridClock initialized: now 1790407747427246 us; error 31 us; skew 500 ppm
I20260926 07:29:07.429056 20220 webserver.cc:460] Webserver started at http://127.0.98.1:42727/ using document root <none> and password file <none>
I20260926 07:29:07.429623 20220 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:07.429711 20220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:07.429940 20220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:07.432024 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/data/instance:
uuid: "7260083d3e684a10a0087f2f8d5a62fb"
format_stamp: "Formatted at 2026-09-26 07:29:07 on dist-test-slave-g04g"
server_key: "e87f59da0a6b86e63fbee9cb653fb7f5"
server_key_iv: "973f266afc0e7ebdd24d69d869b9d9a5"
server_key_version: "encryptionkey@0"
I20260926 07:29:07.432555 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7260083d3e684a10a0087f2f8d5a62fb"
format_stamp: "Formatted at 2026-09-26 07:29:07 on dist-test-slave-g04g"
server_key: "e87f59da0a6b86e63fbee9cb653fb7f5"
server_key_iv: "973f266afc0e7ebdd24d69d869b9d9a5"
server_key_version: "encryptionkey@0"
I20260926 07:29:07.436044 20220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:29:07.438524 20234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:07.439783 20220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:07.439919 20220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/wal
uuid: "7260083d3e684a10a0087f2f8d5a62fb"
format_stamp: "Formatted at 2026-09-26 07:29:07 on dist-test-slave-g04g"
server_key: "e87f59da0a6b86e63fbee9cb653fb7f5"
server_key_iv: "973f266afc0e7ebdd24d69d869b9d9a5"
server_key_version: "encryptionkey@0"
I20260926 07:29:07.440029 20220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:07.459810 20220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:07.460469 20220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:07.460660 20220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:07.461222 20220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:07.462129 20220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:07.462200 20220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:07.462268 20220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:07.462323 20220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:07.471427 20220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39035
I20260926 07:29:07.471457 20347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39035 every 8 connection(s)
I20260926 07:29:07.472290 20220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/data/info.pb
I20260926 07:29:07.478994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 20220
I20260926 07:29:07.479136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-0/wal/instance
I20260926 07:29:07.479461 392 external_mini_cluster.cc:1428] Setting key c25573f02041accc1594c3e14f159ddf
I20260926 07:29:07.481504 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.474047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42159 (local address 127.0.98.62:39725)
0926 07:29:07.474183 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:07.474186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:07.475280 (+ 1094us) server_negotiation.cc:407] Connection header received
0926 07:29:07.476577 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:07.476580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:07.476624 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:07.476699 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:07.477547 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.478069 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.479448 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.479723 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.479803 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.481040 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:07.481044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:07.481047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:07.481065 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:07.481083 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:07.481091 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:07.481150 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:07.481350 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 07:29:07.481398 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:29:07.481662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39725
--builtin_ntp_servers=127.0.98.20:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:29:07.482697 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.474446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39725 (local address 127.0.98.1:42159)
0926 07:29:07.475118 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:07.475159 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:07.476389 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:07.476854 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:07.476860 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:07.476941 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:07.477424 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.477436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.478169 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.478171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.479336 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.479346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.479872 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.479879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.480055 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.480728 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:29:07.480730 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:07.480784 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:29:07.480863 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:07.481176 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:07.481195 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:07.481202 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:29:07.481346 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 07:29:07.481642 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":106,"threads_started":1}
I20260926 07:29:07.483944 20348 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39725
I20260926 07:29:07.484221 20348 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:07.484818 20348 heartbeater.cc:511] Master 127.0.98.62:39725 requested a full tablet report, sending...
I20260926 07:29:07.486341 20160 ts_manager.cc:194] Registered new tserver with Master: 7260083d3e684a10a0087f2f8d5a62fb (127.0.98.1:39035)
I20260926 07:29:07.487540 20160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42159
W20260926 07:29:07.592549 20351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:07.592831 20351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:07.592911 20351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:07.596917 20351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:07.596992 20351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:07.597079 20351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 07:29:07.601792 20351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:07.602888 20351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:07.604305 20351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:07.613909 20357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:07.613974 20359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:07.613899 20356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:07.614130 20351 server_base.cc:1034] running on GCE node
I20260926 07:29:07.614544 20351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:07.615195 20351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:07.616359 20351 hybrid_clock.cc:648] HybridClock initialized: now 1790407747616337 us; error 26 us; skew 500 ppm
I20260926 07:29:07.618080 20351 webserver.cc:460] Webserver started at http://127.0.98.2:36511/ using document root <none> and password file <none>
I20260926 07:29:07.618635 20351 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:07.618693 20351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:07.618927 20351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:07.621141 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/data/instance:
uuid: "910a148da6d8465694fc39becc23207f"
format_stamp: "Formatted at 2026-09-26 07:29:07 on dist-test-slave-g04g"
server_key: "142684b9e4f71559c2cb370f941dc397"
server_key_iv: "d7c644c7be3eff3d99f3ada5e4327790"
server_key_version: "encryptionkey@0"
I20260926 07:29:07.621685 20351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/wal/instance:
uuid: "910a148da6d8465694fc39becc23207f"
format_stamp: "Formatted at 2026-09-26 07:29:07 on dist-test-slave-g04g"
server_key: "142684b9e4f71559c2cb370f941dc397"
server_key_iv: "d7c644c7be3eff3d99f3ada5e4327790"
server_key_version: "encryptionkey@0"
I20260926 07:29:07.625181 20351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:29:07.627565 20365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:07.628747 20351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:07.628877 20351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/wal
uuid: "910a148da6d8465694fc39becc23207f"
format_stamp: "Formatted at 2026-09-26 07:29:07 on dist-test-slave-g04g"
server_key: "142684b9e4f71559c2cb370f941dc397"
server_key_iv: "d7c644c7be3eff3d99f3ada5e4327790"
server_key_version: "encryptionkey@0"
I20260926 07:29:07.628986 20351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:07.639230 20351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:07.639812 20351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:07.639997 20351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:07.640523 20351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:07.641389 20351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:07.641458 20351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:07.641525 20351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:07.641559 20351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:07.650614 20351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39245
I20260926 07:29:07.650683 20478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39245 every 8 connection(s)
I20260926 07:29:07.651607 20351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/data/info.pb
I20260926 07:29:07.658293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 20351
I20260926 07:29:07.658437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/wal/instance
I20260926 07:29:07.658775 392 external_mini_cluster.cc:1428] Setting key 3e0cae93cedd3f73e8e11d25be37e9bd
I20260926 07:29:07.660638 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.653348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33119 (local address 127.0.98.62:39725)
0926 07:29:07.653487 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:07.653490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:07.654479 (+ 989us) server_negotiation.cc:407] Connection header received
0926 07:29:07.655907 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:07.655910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:07.655954 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:07.656026 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:07.656826 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.657329 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.658693 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.658948 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.659067 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.660176 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:07.660179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:07.660182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:07.660196 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:07.660219 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:07.660228 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:07.660297 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:07.660475 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 07:29:07.660521 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 07:29:07.660868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39725
--builtin_ntp_servers=127.0.98.20:46453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:29:07.661801 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.653758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39725 (local address 127.0.98.2:33119)
0926 07:29:07.654360 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:07.654398 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:07.655700 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:07.656160 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:07.656166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:07.656247 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:07.656715 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.656726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.657441 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.657443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.658575 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.658589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.659129 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.659134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.659320 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.659943 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:29:07.659944 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:07.659984 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:29:07.660044 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:07.660322 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:07.660332 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:07.660338 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:29:07.660472 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 07:29:07.660764 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260926 07:29:07.662935 20479 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39725
I20260926 07:29:07.663208 20479 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:07.663748 20479 heartbeater.cc:511] Master 127.0.98.62:39725 requested a full tablet report, sending...
I20260926 07:29:07.664806 20160 ts_manager.cc:194] Registered new tserver with Master: 910a148da6d8465694fc39becc23207f (127.0.98.2:39245)
I20260926 07:29:07.665390 20160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:33119
W20260926 07:29:07.769577 20482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:07.769841 20482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:07.769906 20482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:07.774034 20482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:07.774101 20482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:07.774185 20482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 07:29:07.778744 20482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:07.779904 20482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:07.781281 20482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:07.790537 20488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:07.790614 20487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:07.790663 20482 server_base.cc:1034] running on GCE node
W20260926 07:29:07.790529 20490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:07.791244 20482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:07.791852 20482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:07.793012 20482 hybrid_clock.cc:648] HybridClock initialized: now 1790407747792981 us; error 42 us; skew 500 ppm
I20260926 07:29:07.794786 20482 webserver.cc:460] Webserver started at http://127.0.98.3:43193/ using document root <none> and password file <none>
I20260926 07:29:07.795396 20482 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:07.795460 20482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:07.795688 20482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:07.797734 20482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/data/instance:
uuid: "77d4349e1589481688554206969a3fb6"
format_stamp: "Formatted at 2026-09-26 07:29:07 on dist-test-slave-g04g"
server_key: "f2db74d7a683d789288d890a035668de"
server_key_iv: "0197f41fcaf7d1279534efce33b72bd6"
server_key_version: "encryptionkey@0"
I20260926 07:29:07.798286 20482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/wal/instance:
uuid: "77d4349e1589481688554206969a3fb6"
format_stamp: "Formatted at 2026-09-26 07:29:07 on dist-test-slave-g04g"
server_key: "f2db74d7a683d789288d890a035668de"
server_key_iv: "0197f41fcaf7d1279534efce33b72bd6"
server_key_version: "encryptionkey@0"
I20260926 07:29:07.801891 20482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:29:07.804284 20496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:07.805444 20482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:29:07.805572 20482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/wal
uuid: "77d4349e1589481688554206969a3fb6"
format_stamp: "Formatted at 2026-09-26 07:29:07 on dist-test-slave-g04g"
server_key: "f2db74d7a683d789288d890a035668de"
server_key_iv: "0197f41fcaf7d1279534efce33b72bd6"
server_key_version: "encryptionkey@0"
I20260926 07:29:07.805680 20482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:07.839440 20482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:07.840173 20482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:07.840373 20482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:07.840973 20482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:29:07.841912 20482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:29:07.841992 20482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:07.842060 20482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:29:07.842111 20482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:07.851428 20482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43251
I20260926 07:29:07.851455 20609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43251 every 8 connection(s)
I20260926 07:29:07.852384 20482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/data/info.pb
I20260926 07:29:07.857443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 20482
I20260926 07:29:07.857570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/wal/instance
I20260926 07:29:07.857829 392 external_mini_cluster.cc:1428] Setting key d8f15efd8ca9fda302a7a320297c42f4
I20260926 07:29:07.861689 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.854245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45219 (local address 127.0.98.62:39725)
0926 07:29:07.854382 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:07.854385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:07.855516 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 07:29:07.856892 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:07.856895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:07.856950 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:07.857059 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:07.858031 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.858670 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.859874 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.860105 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.860218 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.861213 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:07.861216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:07.861219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:07.861238 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:07.861255 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:07.861263 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:07.861342 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:07.861497 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 07:29:07.861553 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:29:07.862555 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.854632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39725 (local address 127.0.98.3:45219)
0926 07:29:07.855359 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:07.855400 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:07.856688 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:07.857243 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:07.857250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:07.857334 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:07.857865 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.857879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.858796 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.858799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.859749 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.859759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.860252 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.860255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.860420 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.860972 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:29:07.860973 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:07.861012 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:29:07.861069 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:07.861345 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:07.861356 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:07.861363 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 07:29:07.861497 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 07:29:07.861788 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260926 07:29:07.863593 20610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39725
I20260926 07:29:07.863826 20610 heartbeater.cc:464] Registering TS with master...
I20260926 07:29:07.864324 20610 heartbeater.cc:511] Master 127.0.98.62:39725 requested a full tablet report, sending...
I20260926 07:29:07.865267 20160 ts_manager.cc:194] Registered new tserver with Master: 77d4349e1589481688554206969a3fb6 (127.0.98.3:43251)
I20260926 07:29:07.865810 20160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45219
I20260926 07:29:07.871605 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:29:07.877275 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.873439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.0.98.62:39725)
0926 07:29:07.873579 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:07.873583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:07.873782 (+ 199us) server_negotiation.cc:407] Connection header received
0926 07:29:07.873873 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:07.873875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:07.873911 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:07.873981 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:07.874178 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.874659 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.875748 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.876037 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.876137 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.876780 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:07.876783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:07.876785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:07.876797 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:07.876813 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:07.876863 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:07.876929 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:07.877021 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 07:29:07.877070 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:29:07.880090 20160 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:29:07.883002 20160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:29:07.896767 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.890909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39035 (local address 127.0.0.1:58612)
0926 07:29:07.891610 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:07.891658 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:07.891784 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:07.892548 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:07.892555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:07.892577 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:07.892708 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.892715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.893885 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.893891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.895175 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.895185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.896213 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.896216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.896313 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.896347 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:29:07.896456 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:29:07.896593 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":63,"thread_start_us":75,"threads_started":1}
I20260926 07:29:07.896973 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.891338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.0.98.2:39245)
0926 07:29:07.891680 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:07.891695 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:07.892036 (+ 341us) server_negotiation.cc:407] Connection header received
0926 07:29:07.892125 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:07.892133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:07.892512 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:07.892946 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:07.893089 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.894260 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.895639 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.896375 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.896473 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.896633 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:07.896697 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 07:29:07.896792 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":115,"threads_started":1}
I20260926 07:29:07.897116 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.891949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.0.98.3:43251)
0926 07:29:07.892352 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:07.892363 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:07.892715 (+ 352us) server_negotiation.cc:407] Connection header received
0926 07:29:07.892787 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:07.892793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:07.892958 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:07.893102 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:07.893308 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.894368 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.895561 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.896116 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.896240 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.896417 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:07.896830 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 07:29:07.896944 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":190,"threads_started":1}
I20260926 07:29:07.897382 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.891645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43251 (local address 127.0.0.1:42260)
0926 07:29:07.892512 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:07.892525 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:07.892620 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:07.893084 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:07.893087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:07.893105 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:07.893193 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.893199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.894476 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.894482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.895417 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.895425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.896476 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.896479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.896601 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.896621 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:29:07.896687 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 07:29:07.896742 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":28,"thread_start_us":84,"threads_started":1}
I20260926 07:29:07.897382 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.891259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39245 (local address 127.0.0.1:54190)
0926 07:29:07.891920 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:07.891933 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:07.892026 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:07.892708 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:07.892710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:07.892726 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:07.892816 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.892822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.894383 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.894386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.895517 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.895526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.896482 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.896485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.896573 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.896600 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 07:29:07.896658 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 07:29:07.896726 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260926 07:29:07.897506 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.891083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.0.98.1:39035)
0926 07:29:07.891792 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:07.891803 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:07.891831 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:29:07.891912 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:07.891918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:07.892175 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:07.892344 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:07.892846 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.893773 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.895306 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.896084 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.896195 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.896387 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:07.896832 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 07:29:07.897341 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":128,"threads_started":1}
I20260926 07:29:07.900321 20413 tablet_service.cc:1467] Processing CreateTablet for tablet 80b9a61a61a54d25baea6ba9fc821fce (DEFAULT_TABLE table=test-table [id=45d8403adb914a22af8fd336372a4ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:07.900319 20544 tablet_service.cc:1467] Processing CreateTablet for tablet 80b9a61a61a54d25baea6ba9fc821fce (DEFAULT_TABLE table=test-table [id=45d8403adb914a22af8fd336372a4ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:07.901448 20413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b9a61a61a54d25baea6ba9fc821fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:07.901448 20544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b9a61a61a54d25baea6ba9fc821fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:07.904786 20282 tablet_service.cc:1467] Processing CreateTablet for tablet 80b9a61a61a54d25baea6ba9fc821fce (DEFAULT_TABLE table=test-table [id=45d8403adb914a22af8fd336372a4ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:29:07.905658 20282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b9a61a61a54d25baea6ba9fc821fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:07.908773 20627 tablet_bootstrap.cc:492] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6: Bootstrap starting.
I20260926 07:29:07.909193 20628 tablet_bootstrap.cc:492] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f: Bootstrap starting.
I20260926 07:29:07.911649 20628 tablet_bootstrap.cc:654] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:07.912518 20629 tablet_bootstrap.cc:492] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb: Bootstrap starting.
I20260926 07:29:07.912518 20628 log.cc:824] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:07.912905 20627 tablet_bootstrap.cc:654] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:07.913707 20627 log.cc:824] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:07.914695 20629 tablet_bootstrap.cc:654] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:07.914722 20628 tablet_bootstrap.cc:492] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f: No bootstrap required, opened a new log
I20260926 07:29:07.914925 20628 ts_tablet_manager.cc:1397] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:29:07.915479 20629 log.cc:824] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:07.915671 20627 tablet_bootstrap.cc:492] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6: No bootstrap required, opened a new log
I20260926 07:29:07.915855 20627 ts_tablet_manager.cc:1397] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 07:29:07.917795 20629 tablet_bootstrap.cc:492] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb: No bootstrap required, opened a new log
I20260926 07:29:07.917976 20629 ts_tablet_manager.cc:1397] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 07:29:07.918303 20628 raft_consensus.cc:348] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260083d3e684a10a0087f2f8d5a62fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "910a148da6d8465694fc39becc23207f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39245 } } peers { permanent_uuid: "77d4349e1589481688554206969a3fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43251 } }
I20260926 07:29:07.918547 20628 raft_consensus.cc:374] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:07.918601 20628 raft_consensus.cc:729] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910a148da6d8465694fc39becc23207f, State: Initialized, Role: FOLLOWER
I20260926 07:29:07.919081 20628 consensus_queue.cc:260] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260083d3e684a10a0087f2f8d5a62fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "910a148da6d8465694fc39becc23207f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39245 } } peers { permanent_uuid: "77d4349e1589481688554206969a3fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43251 } }
I20260926 07:29:07.919677 20479 heartbeater.cc:503] Master 127.0.98.62:39725 was elected leader, sending a full tablet report...
I20260926 07:29:07.920269 20627 raft_consensus.cc:348] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260083d3e684a10a0087f2f8d5a62fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "910a148da6d8465694fc39becc23207f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39245 } } peers { permanent_uuid: "77d4349e1589481688554206969a3fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43251 } }
I20260926 07:29:07.920523 20627 raft_consensus.cc:374] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:07.920570 20627 raft_consensus.cc:729] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77d4349e1589481688554206969a3fb6, State: Initialized, Role: FOLLOWER
I20260926 07:29:07.921125 20628 ts_tablet_manager.cc:1428] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 07:29:07.921200 20627 consensus_queue.cc:260] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260083d3e684a10a0087f2f8d5a62fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "910a148da6d8465694fc39becc23207f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39245 } } peers { permanent_uuid: "77d4349e1589481688554206969a3fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43251 } }
I20260926 07:29:07.921842 20610 heartbeater.cc:503] Master 127.0.98.62:39725 was elected leader, sending a full tablet report...
I20260926 07:29:07.921904 20629 raft_consensus.cc:348] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260083d3e684a10a0087f2f8d5a62fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "910a148da6d8465694fc39becc23207f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39245 } } peers { permanent_uuid: "77d4349e1589481688554206969a3fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43251 } }
I20260926 07:29:07.922127 20627 ts_tablet_manager.cc:1428] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 07:29:07.922156 20629 raft_consensus.cc:374] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:07.922209 20629 raft_consensus.cc:729] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7260083d3e684a10a0087f2f8d5a62fb, State: Initialized, Role: FOLLOWER
I20260926 07:29:07.922727 20629 consensus_queue.cc:260] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260083d3e684a10a0087f2f8d5a62fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "910a148da6d8465694fc39becc23207f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39245 } } peers { permanent_uuid: "77d4349e1589481688554206969a3fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43251 } }
I20260926 07:29:07.923439 20348 heartbeater.cc:503] Master 127.0.98.62:39725 was elected leader, sending a full tablet report...
I20260926 07:29:07.923843 20629 ts_tablet_manager.cc:1428] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260926 07:29:07.961467 20633 raft_consensus.cc:482] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:29:07.961685 20633 raft_consensus.cc:504] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260083d3e684a10a0087f2f8d5a62fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "910a148da6d8465694fc39becc23207f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39245 } } peers { permanent_uuid: "77d4349e1589481688554206969a3fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43251 } }
I20260926 07:29:07.962811 20633 leader_election.cc:288] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7260083d3e684a10a0087f2f8d5a62fb (127.0.98.1:39035), 77d4349e1589481688554206969a3fb6 (127.0.98.3:43251)
I20260926 07:29:07.966507 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.963178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38609 (local address 127.0.98.1:39035)
0926 07:29:07.963318 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:07.963322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:07.963336 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:29:07.963501 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:07.963507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:07.963565 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:07.963648 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:07.963907 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.964462 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.965598 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.965973 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.966088 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.966147 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:07.966319 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 07:29:07.966381 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:29:07.966501 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.963085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39035 (local address 127.0.98.2:38609)
0926 07:29:07.963240 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:07.963255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:07.963352 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:07.963657 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:07.963660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:07.963675 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:07.963786 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.963795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.964578 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.964583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.965481 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.965491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.966089 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.966092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.966187 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.966209 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:29:07.966303 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 07:29:07.966373 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":6}
I20260926 07:29:07.966890 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.963477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57031 (local address 127.0.98.3:43251)
0926 07:29:07.963618 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:07.963623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:07.963821 (+ 198us) server_negotiation.cc:407] Connection header received
0926 07:29:07.963942 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:07.963944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:07.963991 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:07.964075 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:29:07.964341 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.964871 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.965915 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.966298 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.966410 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.966462 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:07.966717 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 07:29:07.966782 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:29:07.967192 20302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b9a61a61a54d25baea6ba9fc821fce" candidate_uuid: "910a148da6d8465694fc39becc23207f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7260083d3e684a10a0087f2f8d5a62fb" is_pre_election: true
I20260926 07:29:07.967315 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.963377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43251 (local address 127.0.98.2:57031)
0926 07:29:07.963743 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:07.963753 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:07.963857 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:07.964084 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:07.964087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:07.964101 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:29:07.964238 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.964243 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.964975 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.964980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.965802 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:07.965811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.966419 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.966422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:07.966512 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.966547 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:29:07.967106 (+ 559us) client_negotiation.cc:240] Negotiation successful
0926 07:29:07.967174 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":101,"threads_started":1}
I20260926 07:29:07.967530 20302 raft_consensus.cc:2454] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910a148da6d8465694fc39becc23207f in term 0.
I20260926 07:29:07.967840 20564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b9a61a61a54d25baea6ba9fc821fce" candidate_uuid: "910a148da6d8465694fc39becc23207f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77d4349e1589481688554206969a3fb6" is_pre_election: true
I20260926 07:29:07.967989 20367 leader_election.cc:302] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7260083d3e684a10a0087f2f8d5a62fb, 910a148da6d8465694fc39becc23207f; no voters:
I20260926 07:29:07.968127 20564 raft_consensus.cc:2454] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910a148da6d8465694fc39becc23207f in term 0.
I20260926 07:29:07.968263 20633 raft_consensus.cc:2781] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:29:07.968350 20633 raft_consensus.cc:482] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:29:07.968410 20633 raft_consensus.cc:3037] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:07.969412 20633 raft_consensus.cc:504] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260083d3e684a10a0087f2f8d5a62fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "910a148da6d8465694fc39becc23207f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39245 } } peers { permanent_uuid: "77d4349e1589481688554206969a3fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43251 } }
I20260926 07:29:07.969724 20633 leader_election.cc:288] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [CANDIDATE]: Term 1 election: Requested vote from peers 7260083d3e684a10a0087f2f8d5a62fb (127.0.98.1:39035), 77d4349e1589481688554206969a3fb6 (127.0.98.3:43251)
I20260926 07:29:07.969995 20302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b9a61a61a54d25baea6ba9fc821fce" candidate_uuid: "910a148da6d8465694fc39becc23207f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7260083d3e684a10a0087f2f8d5a62fb"
I20260926 07:29:07.970101 20564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b9a61a61a54d25baea6ba9fc821fce" candidate_uuid: "910a148da6d8465694fc39becc23207f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77d4349e1589481688554206969a3fb6"
I20260926 07:29:07.970118 20302 raft_consensus.cc:3037] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:07.970232 20564 raft_consensus.cc:3037] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:07.971428 20564 raft_consensus.cc:2454] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910a148da6d8465694fc39becc23207f in term 1.
I20260926 07:29:07.971428 20302 raft_consensus.cc:2454] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910a148da6d8465694fc39becc23207f in term 1.
I20260926 07:29:07.971732 20367 leader_election.cc:302] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77d4349e1589481688554206969a3fb6, 910a148da6d8465694fc39becc23207f; no voters:
I20260926 07:29:07.971885 20633 raft_consensus.cc:2781] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:07.972108 20633 raft_consensus.cc:686] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [term 1 LEADER]: Becoming Leader. State: Replica: 910a148da6d8465694fc39becc23207f, State: Running, Role: LEADER
I20260926 07:29:07.972431 20633 consensus_queue.cc:237] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260083d3e684a10a0087f2f8d5a62fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } } peers { permanent_uuid: "910a148da6d8465694fc39becc23207f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39245 } } peers { permanent_uuid: "77d4349e1589481688554206969a3fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43251 } }
W20260926 07:29:07.973332 20349 tablet.cc:2369] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:29:07.975471 20160 catalog_manager.cc:5495] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f reported cstate change: term changed from 0 to 1, leader changed from <none> to 910a148da6d8465694fc39becc23207f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "910a148da6d8465694fc39becc23207f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7260083d3e684a10a0087f2f8d5a62fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "910a148da6d8465694fc39becc23207f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77d4349e1589481688554206969a3fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43251 } health_report { overall_health: UNKNOWN } } }
I20260926 07:29:07.994784 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:07.991271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.0.98.2:39245)
0926 07:29:07.991404 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:07.991409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:07.991423 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:29:07.991616 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:07.991618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:07.991671 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:07.991780 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:07.992026 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.992518 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.993372 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:07.993592 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:07.993703 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:07.993991 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:07.994388 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:07.994517 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:07.994579 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 07:29:07.994628 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:29:08.000274 20564 raft_consensus.cc:1261] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6 [term 1 FOLLOWER]: Refusing update from remote peer 910a148da6d8465694fc39becc23207f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:08.000310 20302 raft_consensus.cc:1261] T 80b9a61a61a54d25baea6ba9fc821fce P 7260083d3e684a10a0087f2f8d5a62fb [term 1 FOLLOWER]: Refusing update from remote peer 910a148da6d8465694fc39becc23207f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:08.000928 20633 consensus_queue.cc:1035] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [LEADER]: Connected to new peer: Peer: permanent_uuid: "77d4349e1589481688554206969a3fb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:08.001119 20637 consensus_queue.cc:1035] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7260083d3e684a10a0087f2f8d5a62fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:08.007535 20640 mvcc.cc:205] Tried to move back new op lower bound from 7333510135801282560 to 7333510135697039360. Current Snapshot: MvccSnapshot[applied={T|T < 7333510135801282560}]
I20260926 07:29:08.010150 20642 mvcc.cc:205] Tried to move back new op lower bound from 7333510135801282560 to 7333510135697039360. Current Snapshot: MvccSnapshot[applied={T|T < 7333510135801282560}]
I20260926 07:29:08.016755 20159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:08.016884 20159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:29:08.019071 20159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 803fe952d8014f118c7dd7d3da8099e6: Sending DeleteTablet for 3 replicas of tablet 80b9a61a61a54d25baea6ba9fc821fce
I20260926 07:29:08.019871 20282 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b9a61a61a54d25baea6ba9fc821fce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:08 UTC) from {username='slave'} at 127.0.0.1:58612
I20260926 07:29:08.019827 20413 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b9a61a61a54d25baea6ba9fc821fce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:08 UTC) from {username='slave'} at 127.0.0.1:54190
I20260926 07:29:08.020128 20544 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b9a61a61a54d25baea6ba9fc821fce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:29:08 UTC) from {username='slave'} at 127.0.0.1:42260
I20260926 07:29:08.020354 20651 tablet_replica.cc:330] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f: stopping tablet replica
I20260926 07:29:08.020632 20651 raft_consensus.cc:2229] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:29:08.020905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 20220
I20260926 07:29:08.020936 20651 raft_consensus.cc:2258] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:08.022186 20652 tablet_replica.cc:330] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6: stopping tablet replica
I20260926 07:29:08.022464 20652 raft_consensus.cc:2229] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:29:08.022737 20652 raft_consensus.cc:2258] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:29:08.023290 20651 ts_tablet_manager.cc:1905] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:08.024159 20652 ts_tablet_manager.cc:1905] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:29:08.026129 20651 ts_tablet_manager.cc:1918] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:08.026286 20651 log.cc:1196] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-1/wal/wals/80b9a61a61a54d25baea6ba9fc821fce
I20260926 07:29:08.026652 20651 ts_tablet_manager.cc:1939] T 80b9a61a61a54d25baea6ba9fc821fce P 910a148da6d8465694fc39becc23207f: Deleting consensus metadata
I20260926 07:29:08.027379 20652 ts_tablet_manager.cc:1918] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:29:08.027474 20652 log.cc:1196] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6: Deleting WAL directory at /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790407489657243-392-0/minicluster-data/ts-2/wal/wals/80b9a61a61a54d25baea6ba9fc821fce
I20260926 07:29:08.027777 20652 ts_tablet_manager.cc:1939] T 80b9a61a61a54d25baea6ba9fc821fce P 77d4349e1589481688554206969a3fb6: Deleting consensus metadata
I20260926 07:29:08.028061 20146 catalog_manager.cc:4841] TS 910a148da6d8465694fc39becc23207f (127.0.98.2:39245): tablet 80b9a61a61a54d25baea6ba9fc821fce (table test-table [id=45d8403adb914a22af8fd336372a4ace]) successfully deleted
W20260926 07:29:08.028545 20145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39035 (error 108)
I20260926 07:29:08.028717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 20351
I20260926 07:29:08.028966 20145 catalog_manager.cc:4841] TS 77d4349e1589481688554206969a3fb6 (127.0.98.3:43251): tablet 80b9a61a61a54d25baea6ba9fc821fce (table test-table [id=45d8403adb914a22af8fd336372a4ace]) successfully deleted
I20260926 07:29:08.031168 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:08.030832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39035 (local address 127.0.0.1:58616)
0926 07:29:08.030963 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:08.031084 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 07:29:08.031311 20145 catalog_manager.cc:4584] TS 7260083d3e684a10a0087f2f8d5a62fb (127.0.98.1:39035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80b9a61a61a54d25baea6ba9fc821fce: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39035: connect: Connection refused (error 111)
I20260926 07:29:08.034960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 20482
I20260926 07:29:08.040292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 20129
2026-09-26T07:29:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1015 ms)
[----------] 6 tests from EncryptionPolicyTest (10195 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:08 dist-test-slave-g04g krb5kdc[20656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:08 dist-test-slave-g04g krb5kdc[20656](info): set up 2 sockets
Sep 26 07:29:08 dist-test-slave-g04g krb5kdc[20656](info): commencing operation
krb5kdc: starting...
W20260926 07:29:09.163018 392 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:09 dist-test-slave-g04g krb5kdc[20656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:29:09.286008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:09.394376 20669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:09.394626 20669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:09.394665 20669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:09.398597 20669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:09.398654 20669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:09.398674 20669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:09.398689 20669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:09.398753 20669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:09.403997 20669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:09.405012 20669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:09.406030 20669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:09.414593 20677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:09.414628 20674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:09.414628 20669 server_base.cc:1034] running on GCE node
W20260926 07:29:09.414867 20675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:09.415443 20669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:09.416543 20669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:09.417744 20669 hybrid_clock.cc:648] HybridClock initialized: now 1790407749417732 us; error 24 us; skew 500 ppm
Sep 26 07:29:09 dist-test-slave-g04g krb5kdc[20656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:09.420521 20669 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:29:09.421391 20669 webserver.cc:460] Webserver started at http://127.0.0.1:44799/ using document root <none> and password file <none>
I20260926 07:29:09.421947 20669 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:09.422039 20669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:09.422271 20669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:09.424382 20669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "b98355f0db824a508b8598394ca8f309"
format_stamp: "Formatted at 2026-09-26 07:29:09 on dist-test-slave-g04g"
server_key: "017c02fef49a1da5047f37a55a23ceed"
server_key_iv: "7ce08bb1667fcd569686cad3bd3e390b"
server_key_version: "encryptionkey@0"
I20260926 07:29:09.424948 20669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "b98355f0db824a508b8598394ca8f309"
format_stamp: "Formatted at 2026-09-26 07:29:09 on dist-test-slave-g04g"
server_key: "017c02fef49a1da5047f37a55a23ceed"
server_key_iv: "7ce08bb1667fcd569686cad3bd3e390b"
server_key_version: "encryptionkey@0"
I20260926 07:29:09.428614 20669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:29:09.430930 20684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:09.432335 20669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:09.432492 20669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "b98355f0db824a508b8598394ca8f309"
format_stamp: "Formatted at 2026-09-26 07:29:09 on dist-test-slave-g04g"
server_key: "017c02fef49a1da5047f37a55a23ceed"
server_key_iv: "7ce08bb1667fcd569686cad3bd3e390b"
server_key_version: "encryptionkey@0"
I20260926 07:29:09.432612 20669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:09.455652 20669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:09.458604 20669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:09.458817 20669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:09.465811 20669 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46803
I20260926 07:29:09.465837 20736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46803 every 8 connection(s)
I20260926 07:29:09.466745 20669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:09.469439 20737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:09.472123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 20669
I20260926 07:29:09.472270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:09.472566 392 external_mini_cluster.cc:1428] Setting key 2b5628d4deb0378f2e551d8f7009e4c7
I20260926 07:29:09.475313 20737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309: Bootstrap starting.
I20260926 07:29:09.477483 20737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:09.478199 20737 log.cc:824] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:09.480154 20737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309: No bootstrap required, opened a new log
Sep 26 07:29:09 dist-test-slave-g04g krb5kdc[20656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:29:09.483771 20737 raft_consensus.cc:348] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98355f0db824a508b8598394ca8f309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46803 } }
I20260926 07:29:09.483976 20737 raft_consensus.cc:374] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:09.484033 20737 raft_consensus.cc:729] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98355f0db824a508b8598394ca8f309, State: Initialized, Role: FOLLOWER
I20260926 07:29:09.484499 20737 consensus_queue.cc:260] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98355f0db824a508b8598394ca8f309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46803 } }
I20260926 07:29:09.484642 20737 raft_consensus.cc:388] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:09.484705 20737 raft_consensus.cc:482] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:09.484819 20737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:09.485636 20737 raft_consensus.cc:504] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98355f0db824a508b8598394ca8f309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46803 } }
I20260926 07:29:09.485960 20737 leader_election.cc:302] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b98355f0db824a508b8598394ca8f309; no voters:
I20260926 07:29:09.486330 20737 leader_election.cc:288] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:09.486438 20742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:09.486655 20742 raft_consensus.cc:686] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [term 1 LEADER]: Becoming Leader. State: Replica: b98355f0db824a508b8598394ca8f309, State: Running, Role: LEADER
I20260926 07:29:09.486997 20742 consensus_queue.cc:237] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98355f0db824a508b8598394ca8f309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46803 } }
I20260926 07:29:09.487581 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:09.473954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58223 (local address 127.0.0.1:46803)
0926 07:29:09.474395 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:09.474409 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:09.474444 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:29:09.475109 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:09.475129 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:09.475510 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:09.475840 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:09.476638 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:09.477663 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:09.478576 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:09.479087 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:09.479274 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:09.482008 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:09.482013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:09.482025 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:09.482051 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:09.484385 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:09.484957 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:09.484964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:09.484969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:09.485082 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:09.485431 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:09.485434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:09.485436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:09.485872 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:09.486090 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:09.486424 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 07:29:09.486653 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":146,"threads_started":1}
I20260926 07:29:09.487609 20737 sys_catalog.cc:564] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:09.488729 20744 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b98355f0db824a508b8598394ca8f309. Latest consensus state: current_term: 1 leader_uuid: "b98355f0db824a508b8598394ca8f309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98355f0db824a508b8598394ca8f309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46803 } } }
I20260926 07:29:09.488850 20744 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:09.489140 20743 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b98355f0db824a508b8598394ca8f309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98355f0db824a508b8598394ca8f309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46803 } } }
I20260926 07:29:09.489223 20743 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:09.489337 20751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:09.490820 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:09.492589 20751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:09.498260 20751 catalog_manager.cc:1347] Generated new cluster ID: aa1c1936e2c4469e90abb65df4572dc9
I20260926 07:29:09.498333 20751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:09.502010 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:09.493113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53305 (local address 127.0.0.1:46803)
0926 07:29:09.493202 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:09.493206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:09.493601 (+ 395us) server_negotiation.cc:407] Connection header received
0926 07:29:09.493780 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:09.493784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:09.493833 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:09.493916 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:09.494667 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:09.495249 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:09.496030 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:09.496273 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:09.496381 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:09.498055 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:09.498060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:09.498062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:09.498087 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:09.499786 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:09.500352 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:09.500356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:09.500358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:09.500437 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:09.500827 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:09.500830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:09.500832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:09.501068 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:09.501218 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:09.501687 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 07:29:09.501828 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:29:09.511113 20751 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:09.512315 20751 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:09.519177 20751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b98355f0db824a508b8598394ca8f309: Generated new TSK 0
I20260926 07:29:09.519937 20751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:09.523849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 20669
W20260926 07:29:09.529273 20654 connection.cc:535] client connection to 127.0.0.1:46803 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46803): Connection reset by peer (error 104)
2026-09-26T07:29:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:09 dist-test-slave-g04g krb5kdc[20768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:09 dist-test-slave-g04g krb5kdc[20768](info): set up 2 sockets
Sep 26 07:29:09 dist-test-slave-g04g krb5kdc[20768](info): commencing operation
krb5kdc: starting...
W20260926 07:29:10.559368 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:10 dist-test-slave-g04g krb5kdc[20768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:29:10.694676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:10.805292 20781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:10.805536 20781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:10.805577 20781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:10.809520 20781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:10.809574 20781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:10.809594 20781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:10.809610 20781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:10.809625 20781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:10.814689 20781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:10.815711 20781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:10.816644 20781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:10.824748 20786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:10.824728 20787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:10.824728 20789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:10.825165 20781 server_base.cc:1034] running on GCE node
I20260926 07:29:10.825613 20781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:10.826627 20781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:10.827845 20781 hybrid_clock.cc:648] HybridClock initialized: now 1790407750827825 us; error 33 us; skew 500 ppm
Sep 26 07:29:10 dist-test-slave-g04g krb5kdc[20768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:10.830379 20781 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:29:10.831210 20781 webserver.cc:460] Webserver started at http://127.0.0.1:41135/ using document root <none> and password file <none>
I20260926 07:29:10.831705 20781 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:10.831759 20781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:10.831907 20781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:10.833808 20781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "f4c0fe67fb464c6cb46651341577bfde"
format_stamp: "Formatted at 2026-09-26 07:29:10 on dist-test-slave-g04g"
server_key: "36485b61e1fe0aa26826adbe3d0ea16d"
server_key_iv: "8a0db748552de4378359b66392085b3d"
server_key_version: "encryptionkey@0"
I20260926 07:29:10.834297 20781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4c0fe67fb464c6cb46651341577bfde"
format_stamp: "Formatted at 2026-09-26 07:29:10 on dist-test-slave-g04g"
server_key: "36485b61e1fe0aa26826adbe3d0ea16d"
server_key_iv: "8a0db748552de4378359b66392085b3d"
server_key_version: "encryptionkey@0"
I20260926 07:29:10.837941 20781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 07:29:10.840216 20796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:10.841364 20781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:29:10.841499 20781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "f4c0fe67fb464c6cb46651341577bfde"
format_stamp: "Formatted at 2026-09-26 07:29:10 on dist-test-slave-g04g"
server_key: "36485b61e1fe0aa26826adbe3d0ea16d"
server_key_iv: "8a0db748552de4378359b66392085b3d"
server_key_version: "encryptionkey@0"
I20260926 07:29:10.841604 20781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:10.856029 20781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:10.858692 20781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:10.858888 20781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:10.865818 20781 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35909
I20260926 07:29:10.865895 20848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35909 every 8 connection(s)
I20260926 07:29:10.866812 20781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:10.869593 20849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:10.870618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 20781
I20260926 07:29:10.870779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:10.871158 392 external_mini_cluster.cc:1428] Setting key 1c62714bcbd42088420c879417248b47
I20260926 07:29:10.875849 20849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde: Bootstrap starting.
I20260926 07:29:10.878343 20849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:10.879276 20849 log.cc:824] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:10.879869 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:10.872447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46057 (local address 127.0.0.1:35909)
0926 07:29:10.872827 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:10.872851 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:10.872885 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:29:10.873523 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:10.873540 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:10.873863 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:10.874136 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:10.874467 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:10.875486 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:10.876640 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:10.877101 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:10.877213 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:10.878058 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:10.878063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:10.878075 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:10.878483 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:10.878546 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:10.878574 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:10.878678 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:10.878887 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 07:29:10.879088 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":124,"threads_started":1}
I20260926 07:29:10.881148 20849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde: No bootstrap required, opened a new log
I20260926 07:29:10.882747 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:10.885483 20849 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c0fe67fb464c6cb46651341577bfde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35909 } }
I20260926 07:29:10.885777 20849 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:10.885845 20849 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c0fe67fb464c6cb46651341577bfde, State: Initialized, Role: FOLLOWER
I20260926 07:29:10.886240 20849 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c0fe67fb464c6cb46651341577bfde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35909 } }
I20260926 07:29:10.886386 20849 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:10.886482 20849 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:10.886577 20849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:10.887596 20849 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c0fe67fb464c6cb46651341577bfde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35909 } }
I20260926 07:29:10.887949 20849 leader_election.cc:302] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4c0fe67fb464c6cb46651341577bfde; no voters:
I20260926 07:29:10.888285 20849 leader_election.cc:288] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:10.888607 20859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:10.888865 20859 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [term 1 LEADER]: Becoming Leader. State: Replica: f4c0fe67fb464c6cb46651341577bfde, State: Running, Role: LEADER
I20260926 07:29:10.889330 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:10.885090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51309 (local address 127.0.0.1:35909)
0926 07:29:10.885201 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:10.885204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:10.885314 (+ 110us) server_negotiation.cc:407] Connection header received
0926 07:29:10.885421 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:10.885423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:10.885457 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:10.885520 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:10.885819 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:10.886498 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:10.887456 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:10.887793 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:10.887900 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:10.888793 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:10.888799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:10.888805 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:10.888821 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:10.888853 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:10.888866 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:10.888935 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:10.889174 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 07:29:10.889207 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:29:10.889364 20859 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c0fe67fb464c6cb46651341577bfde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35909 } }
I20260926 07:29:10.889619 20849 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:10.890954 20861 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c0fe67fb464c6cb46651341577bfde. Latest consensus state: current_term: 1 leader_uuid: "f4c0fe67fb464c6cb46651341577bfde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c0fe67fb464c6cb46651341577bfde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35909 } } }
I20260926 07:29:10.891108 20861 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:10.891261 20860 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4c0fe67fb464c6cb46651341577bfde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c0fe67fb464c6cb46651341577bfde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35909 } } }
I20260926 07:29:10.891348 20860 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde [sys.catalog]: This master's current role is: LEADER
W20260926 07:29:10.894266 20874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:29:10.894330 20874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:10.894411 20870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:10.895169 20870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:10.900370 20870 catalog_manager.cc:1347] Generated new cluster ID: aadaab2713b6458da8d3ec7c121364fd
I20260926 07:29:10.900460 20870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:10.910102 20870 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:10.911123 20870 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:10.918695 20870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4c0fe67fb464c6cb46651341577bfde: Generated new TSK 0
I20260926 07:29:10.919399 20870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:10.930912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 20781
W20260926 07:29:10.935806 20766 connection.cc:535] client connection to 127.0.0.1:35909 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35909): Connection reset by peer (error 104)
2026-09-26T07:29:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:10 dist-test-slave-g04g krb5kdc[20880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:10 dist-test-slave-g04g krb5kdc[20880](info): set up 2 sockets
Sep 26 07:29:10 dist-test-slave-g04g krb5kdc[20880](info): commencing operation
krb5kdc: starting...
W20260926 07:29:11.975544 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:12 dist-test-slave-g04g krb5kdc[20880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:29:12.098670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:12.207710 20893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:12.208016 20893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:12.208104 20893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:12.212083 20893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:12.212183 20893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:12.212222 20893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:12.212260 20893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:12.212289 20893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:12.217540 20893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36567
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:12.218580 20893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:12.219623 20893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:12.228055 20901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:12.228046 20898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:12.228304 20893 server_base.cc:1034] running on GCE node
W20260926 07:29:12.228061 20899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:12.228924 20893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:12.229987 20893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:12.231181 20893 hybrid_clock.cc:648] HybridClock initialized: now 1790407752231126 us; error 75 us; skew 500 ppm
Sep 26 07:29:12 dist-test-slave-g04g krb5kdc[20880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:12.234007 20893 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:29:12.234862 20893 webserver.cc:460] Webserver started at http://127.0.0.1:39551/ using document root <none> and password file <none>
I20260926 07:29:12.235459 20893 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:12.235550 20893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:12.235780 20893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:12.237757 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "7d133cd97c9143769fa47ec66184c2d4"
format_stamp: "Formatted at 2026-09-26 07:29:12 on dist-test-slave-g04g"
server_key: "f420b65df15e63ff46b0bbd097f5f2ad"
server_key_iv: "3cdaf2a7306dba18d0a58f176572ff09"
server_key_version: "encryptionkey@0"
I20260926 07:29:12.238310 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d133cd97c9143769fa47ec66184c2d4"
format_stamp: "Formatted at 2026-09-26 07:29:12 on dist-test-slave-g04g"
server_key: "f420b65df15e63ff46b0bbd097f5f2ad"
server_key_iv: "3cdaf2a7306dba18d0a58f176572ff09"
server_key_version: "encryptionkey@0"
I20260926 07:29:12.241858 20893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:29:12.244218 20908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:12.245383 20893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:29:12.245515 20893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "7d133cd97c9143769fa47ec66184c2d4"
format_stamp: "Formatted at 2026-09-26 07:29:12 on dist-test-slave-g04g"
server_key: "f420b65df15e63ff46b0bbd097f5f2ad"
server_key_iv: "3cdaf2a7306dba18d0a58f176572ff09"
server_key_version: "encryptionkey@0"
I20260926 07:29:12.245623 20893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:12.249967 20893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:12.252817 20893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:12.253005 20893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:12.259750 20893 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36567
I20260926 07:29:12.259774 20960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36567 every 8 connection(s)
I20260926 07:29:12.260684 20893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:12.263391 20961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:12.264508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 20893
I20260926 07:29:12.264683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:12.265044 392 external_mini_cluster.cc:1428] Setting key de0a9c77db7449d56c9a91fabddfd887
I20260926 07:29:12.270147 20961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4: Bootstrap starting.
I20260926 07:29:12.270648 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:12.266430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33079 (local address 127.0.0.1:36567)
0926 07:29:12.266859 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:12.266877 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:12.266919 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:29:12.267781 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:12.267805 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:12.268187 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:12.268557 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:12.269296 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:12.269302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:12.269316 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:12.269328 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:12.269390 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:12.269416 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:12.269560 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:12.269627 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 07:29:12.269774 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260926 07:29:12.272734 20961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:12.273651 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:12.273623 20961 log.cc:824] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:12.275836 20961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4: No bootstrap required, opened a new log
I20260926 07:29:12.277585 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:12.275871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32925 (local address 127.0.0.1:36567)
0926 07:29:12.275998 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:12.276003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:12.276189 (+ 186us) server_negotiation.cc:407] Connection header received
0926 07:29:12.276267 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:12.276270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:12.276306 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:12.276405 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:12.277174 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:12.277178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:12.277181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:12.277196 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:12.277230 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:12.277246 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:12.277348 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:12.277416 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 07:29:12.277457 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:29:12.280321 20961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d133cd97c9143769fa47ec66184c2d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36567 } }
I20260926 07:29:12.280562 20961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:12.280606 20961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d133cd97c9143769fa47ec66184c2d4, State: Initialized, Role: FOLLOWER
I20260926 07:29:12.281054 20961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d133cd97c9143769fa47ec66184c2d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36567 } }
I20260926 07:29:12.281189 20961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:12.281235 20961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:12.281312 20961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:12.282305 20961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d133cd97c9143769fa47ec66184c2d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36567 } }
I20260926 07:29:12.282618 20961 leader_election.cc:302] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d133cd97c9143769fa47ec66184c2d4; no voters:
I20260926 07:29:12.282886 20961 leader_election.cc:288] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:12.283064 20971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:12.283294 20971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [term 1 LEADER]: Becoming Leader. State: Replica: 7d133cd97c9143769fa47ec66184c2d4, State: Running, Role: LEADER
I20260926 07:29:12.283691 20971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d133cd97c9143769fa47ec66184c2d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36567 } }
I20260926 07:29:12.284273 20961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:12.285462 20972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d133cd97c9143769fa47ec66184c2d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d133cd97c9143769fa47ec66184c2d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36567 } } }
I20260926 07:29:12.285610 20972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:12.286059 20980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:12.286048 20973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d133cd97c9143769fa47ec66184c2d4. Latest consensus state: current_term: 1 leader_uuid: "7d133cd97c9143769fa47ec66184c2d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d133cd97c9143769fa47ec66184c2d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36567 } } }
I20260926 07:29:12.286196 20973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:12.289075 20980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:12.294535 20980 catalog_manager.cc:1347] Generated new cluster ID: 2cdd3ee20f604b0aab9ab2454ef65f1f
I20260926 07:29:12.294605 20980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:12.300953 20980 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:12.301290 20980 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:12.311895 20980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d133cd97c9143769fa47ec66184c2d4: Generated new TSK 0
I20260926 07:29:12.312668 20980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:12.321671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 20893
2026-09-26T07:29:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:12 dist-test-slave-g04g krb5kdc[20992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:12 dist-test-slave-g04g krb5kdc[20992](info): set up 2 sockets
Sep 26 07:29:12 dist-test-slave-g04g krb5kdc[20992](info): commencing operation
krb5kdc: starting...
W20260926 07:29:13.359236 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:13 dist-test-slave-g04g krb5kdc[20992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:29:13.483752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:13.592521 21005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:13.592785 21005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:13.592828 21005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:13.596879 21005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:13.596933 21005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:13.596952 21005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:13.596969 21005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:13.596984 21005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:13.602193 21005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:13.603264 21005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:13.604264 21005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:13.612680 21013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:13.612696 21011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:13.612695 21010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:13.613006 21005 server_base.cc:1034] running on GCE node
I20260926 07:29:13.613554 21005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:13.614679 21005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:13.615895 21005 hybrid_clock.cc:648] HybridClock initialized: now 1790407753615863 us; error 48 us; skew 500 ppm
Sep 26 07:29:13 dist-test-slave-g04g krb5kdc[20992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:13.621788 21005 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:29:13.622763 21005 webserver.cc:460] Webserver started at http://127.0.0.1:45203/ using document root <none> and password file <none>
I20260926 07:29:13.623440 21005 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:13.623538 21005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:13.623775 21005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:13.625844 21005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "c1c7c711ed714cb7811826367464a57b"
format_stamp: "Formatted at 2026-09-26 07:29:13 on dist-test-slave-g04g"
server_key: "27761c968508a7e1fedf24f137d8dd5d"
server_key_iv: "4bf9d0782651b0acd2d6e4c1b7259943"
server_key_version: "encryptionkey@0"
I20260926 07:29:13.626425 21005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1c7c711ed714cb7811826367464a57b"
format_stamp: "Formatted at 2026-09-26 07:29:13 on dist-test-slave-g04g"
server_key: "27761c968508a7e1fedf24f137d8dd5d"
server_key_iv: "4bf9d0782651b0acd2d6e4c1b7259943"
server_key_version: "encryptionkey@0"
I20260926 07:29:13.630128 21005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:29:13.632726 21020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:13.634016 21005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:29:13.634145 21005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "c1c7c711ed714cb7811826367464a57b"
format_stamp: "Formatted at 2026-09-26 07:29:13 on dist-test-slave-g04g"
server_key: "27761c968508a7e1fedf24f137d8dd5d"
server_key_iv: "4bf9d0782651b0acd2d6e4c1b7259943"
server_key_version: "encryptionkey@0"
I20260926 07:29:13.634254 21005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:13.659060 21005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:13.662151 21005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:13.662379 21005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:13.669598 21005 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36253
I20260926 07:29:13.669617 21072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36253 every 8 connection(s)
I20260926 07:29:13.670567 21005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:13.673367 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:13.678787 21073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b: Bootstrap starting.
I20260926 07:29:13.680035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 21005
I20260926 07:29:13.680168 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:13.680454 392 external_mini_cluster.cc:1428] Setting key 0d5c36bcaf228dcbd4f50edb1df2f777
I20260926 07:29:13.681563 21073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:13.682312 21073 log.cc:824] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:13.684615 21073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b: No bootstrap required, opened a new log
I20260926 07:29:13.689008 21073 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7c711ed714cb7811826367464a57b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36253 } }
I20260926 07:29:13.689299 21073 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:13.689380 21073 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c7c711ed714cb7811826367464a57b, State: Initialized, Role: FOLLOWER
Sep 26 07:29:13 dist-test-slave-g04g krb5kdc[20992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:29:13.689872 21073 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7c711ed714cb7811826367464a57b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36253 } }
I20260926 07:29:13.690048 21073 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:13.690119 21073 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:13.690213 21073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:13.691495 21073 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7c711ed714cb7811826367464a57b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36253 } }
I20260926 07:29:13.691872 21073 leader_election.cc:302] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1c7c711ed714cb7811826367464a57b; no voters:
I20260926 07:29:13.692215 21073 leader_election.cc:288] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:13.692344 21078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:13.692564 21078 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [term 1 LEADER]: Becoming Leader. State: Replica: c1c7c711ed714cb7811826367464a57b, State: Running, Role: LEADER
I20260926 07:29:13.692922 21078 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7c711ed714cb7811826367464a57b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36253 } }
I20260926 07:29:13.693570 21073 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:13.693967 21078 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1c7c711ed714cb7811826367464a57b. Latest consensus state: current_term: 1 leader_uuid: "c1c7c711ed714cb7811826367464a57b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7c711ed714cb7811826367464a57b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36253 } } }
I20260926 07:29:13.694082 21078 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:13.694404 21080 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1c7c711ed714cb7811826367464a57b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c7c711ed714cb7811826367464a57b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36253 } } }
I20260926 07:29:13.694480 21080 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:13.694500 21082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:13.696422 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:13.681596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49255 (local address 127.0.0.1:36253)
0926 07:29:13.682056 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:13.682072 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:13.682104 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:29:13.682851 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:13.682877 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:13.683294 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:13.683574 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:13.684489 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:13.685498 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:13.686697 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:13.687226 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:13.687343 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:13.690339 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:13.690346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:13.690362 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:13.690397 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:13.692412 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:13.692911 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:13.692917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:13.692922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:13.693046 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:13.693396 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:13.693399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:13.693401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:13.693891 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:13.694816 (+ 925us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:13.694964 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 07:29:13.695333 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":170,"threads_started":1}
I20260926 07:29:13.697439 21082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:13.699461 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:13.703351 21082 catalog_manager.cc:1347] Generated new cluster ID: 9a6c767b5d3c4ac49c9fa58ddebb2af8
I20260926 07:29:13.703473 21082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:13.710556 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:13.701609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47357 (local address 127.0.0.1:36253)
0926 07:29:13.701703 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:13.701707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:13.701945 (+ 238us) server_negotiation.cc:407] Connection header received
0926 07:29:13.702047 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:13.702050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:13.702095 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:13.702181 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:13.702952 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:13.703794 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:13.704921 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:13.705388 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:13.705501 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:13.707026 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:13.707029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:13.707056 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:13.707082 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:13.708772 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:13.709225 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:13.709229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:13.709231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:13.709318 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:13.709650 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:13.709654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:13.709655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:13.709869 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:13.710014 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:13.710225 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 07:29:13.710376 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 07:29:13.716750 21082 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:13.717617 21082 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:13.723914 21082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1c7c711ed714cb7811826367464a57b: Generated new TSK 0
I20260926 07:29:13.724715 21082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:13.728928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 21005
2026-09-26T07:29:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:13 dist-test-slave-g04g krb5kdc[21104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:13 dist-test-slave-g04g krb5kdc[21104](info): set up 2 sockets
Sep 26 07:29:13 dist-test-slave-g04g krb5kdc[21104](info): commencing operation
krb5kdc: starting...
W20260926 07:29:14.799243 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:14 dist-test-slave-g04g krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:29:14.916508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:15.026772 21117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:15.027021 21117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:15.027133 21117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:15.031049 21117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:15.031102 21117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:15.031121 21117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:15.031137 21117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:15.031152 21117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:15.036350 21117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:15.037276 21117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:15.038199 21117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:29:15.046839 21117 server_base.cc:1034] running on GCE node
W20260926 07:29:15.046770 21122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:15.046770 21125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:15.046856 21123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:15.047680 21117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:15.048766 21117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:15.049925 21117 hybrid_clock.cc:648] HybridClock initialized: now 1790407755049906 us; error 65 us; skew 500 ppm
Sep 26 07:29:15 dist-test-slave-g04g krb5kdc[21104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:15.052625 21117 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:29:15.053483 21117 webserver.cc:460] Webserver started at http://127.0.0.1:36037/ using document root <none> and password file <none>
I20260926 07:29:15.054025 21117 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:15.054082 21117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:15.054284 21117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:15.056298 21117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "f3c0a946a40d437db1853bd22db6a9c9"
format_stamp: "Formatted at 2026-09-26 07:29:15 on dist-test-slave-g04g"
server_key: "69656bae8c324a3f2a39d689c235efe0"
server_key_iv: "0c480093965da0b6d93a942d751a8738"
server_key_version: "encryptionkey@0"
I20260926 07:29:15.056841 21117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "f3c0a946a40d437db1853bd22db6a9c9"
format_stamp: "Formatted at 2026-09-26 07:29:15 on dist-test-slave-g04g"
server_key: "69656bae8c324a3f2a39d689c235efe0"
server_key_iv: "0c480093965da0b6d93a942d751a8738"
server_key_version: "encryptionkey@0"
I20260926 07:29:15.060400 21117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:29:15.062618 21132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:15.063942 21117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:15.064088 21117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "f3c0a946a40d437db1853bd22db6a9c9"
format_stamp: "Formatted at 2026-09-26 07:29:15 on dist-test-slave-g04g"
server_key: "69656bae8c324a3f2a39d689c235efe0"
server_key_iv: "0c480093965da0b6d93a942d751a8738"
server_key_version: "encryptionkey@0"
I20260926 07:29:15.064199 21117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:15.074466 21117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:15.077461 21117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:15.077656 21117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:15.084532 21117 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41299
I20260926 07:29:15.084561 21184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41299 every 8 connection(s)
I20260926 07:29:15.085430 21117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:15.088032 21185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:15.092272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 21117
I20260926 07:29:15.092417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:15.092713 392 external_mini_cluster.cc:1428] Setting key 434f4184a61860150013fca3e81fc5ca
I20260926 07:29:15.093775 21185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9: Bootstrap starting.
I20260926 07:29:15.096360 21185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:15.097196 21185 log.cc:824] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:15.099141 21185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9: No bootstrap required, opened a new log
I20260926 07:29:15.102522 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:15.094106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38791 (local address 127.0.0.1:41299)
0926 07:29:15.094517 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:15.094537 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:15.094577 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:29:15.095451 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:15.095473 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:15.095877 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:15.096238 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:15.096706 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:15.098033 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:15.098895 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:15.099597 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:15.099753 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:15.100534 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:15.100540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:15.100555 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:15.101045 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:15.101110 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:15.101138 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:15.101238 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:15.101337 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 07:29:15.101532 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":108,"threads_started":1}
I20260926 07:29:15.103688 21185 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c0a946a40d437db1853bd22db6a9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41299 } }
I20260926 07:29:15.103890 21185 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:15.103924 21185 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c0a946a40d437db1853bd22db6a9c9, State: Initialized, Role: FOLLOWER
I20260926 07:29:15.104430 21185 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c0a946a40d437db1853bd22db6a9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41299 } }
I20260926 07:29:15.104626 21185 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:15.104719 21185 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:15.104822 21185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:15.105500 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:15.105846 21185 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c0a946a40d437db1853bd22db6a9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41299 } }
I20260926 07:29:15.106173 21185 leader_election.cc:302] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3c0a946a40d437db1853bd22db6a9c9; no voters:
I20260926 07:29:15.106460 21185 leader_election.cc:288] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:15.106575 21190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:15.106878 21190 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [term 1 LEADER]: Becoming Leader. State: Replica: f3c0a946a40d437db1853bd22db6a9c9, State: Running, Role: LEADER
I20260926 07:29:15.107219 21190 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c0a946a40d437db1853bd22db6a9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41299 } }
I20260926 07:29:15.107784 21185 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:15.108330 21193 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3c0a946a40d437db1853bd22db6a9c9. Latest consensus state: current_term: 1 leader_uuid: "f3c0a946a40d437db1853bd22db6a9c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c0a946a40d437db1853bd22db6a9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41299 } } }
I20260926 07:29:15.108458 21193 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:15.108724 21195 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3c0a946a40d437db1853bd22db6a9c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c0a946a40d437db1853bd22db6a9c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41299 } } }
I20260926 07:29:15.108841 21195 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9 [sys.catalog]: This master's current role is: LEADER
W20260926 07:29:15.113369 21210 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:29:15.113535 21210 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:15.113662 21200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:15.114531 21200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:15.115104 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:15.107769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41579 (local address 127.0.0.1:41299)
0926 07:29:15.107865 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:15.107869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:15.110355 (+ 2486us) server_negotiation.cc:407] Connection header received
0926 07:29:15.110590 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:15.110595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:15.110649 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:15.110755 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:15.111241 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:15.111824 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:15.113069 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:15.113384 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:15.113508 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:15.114527 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:15.114532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:15.114535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:15.114558 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:15.114608 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:15.114628 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:15.114712 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:15.114829 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 07:29:15.114886 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:29:15.120244 21200 catalog_manager.cc:1347] Generated new cluster ID: 88b05c5f4c5e4c9c8dc604c842856781
I20260926 07:29:15.120311 21200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:15.134882 21200 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:15.135908 21200 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:15.139792 21200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3c0a946a40d437db1853bd22db6a9c9: Generated new TSK 0
I20260926 07:29:15.140651 21200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:15.153790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 21117
2026-09-26T07:29:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:15 dist-test-slave-g04g krb5kdc[21216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:15 dist-test-slave-g04g krb5kdc[21216](info): set up 2 sockets
Sep 26 07:29:15 dist-test-slave-g04g krb5kdc[21216](info): commencing operation
krb5kdc: starting...
W20260926 07:29:16.195740 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:16 dist-test-slave-g04g krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:29:16.312278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44773
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:16.418788 21229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:16.419067 21229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:16.419111 21229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:16.423211 21229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:16.423272 21229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:16.423292 21229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:16.423332 21229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:16.423367 21229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:16.428706 21229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:16.429762 21229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:16.430790 21229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:16.439018 21234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:16.439143 21235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:16.439064 21229 server_base.cc:1034] running on GCE node
W20260926 07:29:16.439083 21237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:16.439821 21229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:16.440903 21229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:16.442116 21229 hybrid_clock.cc:648] HybridClock initialized: now 1790407756442097 us; error 30 us; skew 500 ppm
Sep 26 07:29:16 dist-test-slave-g04g krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:16.444808 21229 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:29:16.445641 21229 webserver.cc:460] Webserver started at http://127.0.0.1:43365/ using document root <none> and password file <none>
I20260926 07:29:16.446175 21229 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:16.446262 21229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:16.446475 21229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:16.448626 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "c51abb0833654130ba034add9a31da3c"
format_stamp: "Formatted at 2026-09-26 07:29:16 on dist-test-slave-g04g"
server_key: "6c8a6bdb25718bc30d08ca59b6a02ceb"
server_key_iv: "54b06717eb04c98c8d3f179514f037be"
server_key_version: "encryptionkey@0"
I20260926 07:29:16.449182 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "c51abb0833654130ba034add9a31da3c"
format_stamp: "Formatted at 2026-09-26 07:29:16 on dist-test-slave-g04g"
server_key: "6c8a6bdb25718bc30d08ca59b6a02ceb"
server_key_iv: "54b06717eb04c98c8d3f179514f037be"
server_key_version: "encryptionkey@0"
I20260926 07:29:16.452672 21229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 07:29:16.454973 21244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:16.456238 21229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:16.456367 21229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "c51abb0833654130ba034add9a31da3c"
format_stamp: "Formatted at 2026-09-26 07:29:16 on dist-test-slave-g04g"
server_key: "6c8a6bdb25718bc30d08ca59b6a02ceb"
server_key_iv: "54b06717eb04c98c8d3f179514f037be"
server_key_version: "encryptionkey@0"
I20260926 07:29:16.456471 21229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:16.460857 21229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:16.463990 21229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:16.464178 21229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:16.470865 21229 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44773
I20260926 07:29:16.470933 21296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44773 every 8 connection(s)
I20260926 07:29:16.471796 21229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:16.474432 21297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:16.477385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 21229
I20260926 07:29:16.477533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:16.477802 392 external_mini_cluster.cc:1428] Setting key 46a041f10f5ba1e92722e0739c8a06c1
I20260926 07:29:16.480408 21297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c: Bootstrap starting.
I20260926 07:29:16.483060 21297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:16.483398 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:16.479191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49999 (local address 127.0.0.1:44773)
0926 07:29:16.479594 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:16.479614 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:16.479656 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:29:16.480466 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:16.480492 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:16.480834 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:16.481178 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:16.482024 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:16.482030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:16.482044 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:16.482056 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:16.482118 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:16.482142 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:16.482279 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:16.482343 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 07:29:16.482483 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":105,"threads_started":1}
I20260926 07:29:16.483922 21297 log.cc:824] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:16.486065 21297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c: No bootstrap required, opened a new log
I20260926 07:29:16.486166 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:16.489866 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:16.488220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48003 (local address 127.0.0.1:44773)
0926 07:29:16.488333 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:16.488337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:16.488477 (+ 140us) server_negotiation.cc:407] Connection header received
0926 07:29:16.488590 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:16.488592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:16.488619 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:16.488682 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:16.489421 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:16.489424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:16.489426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:16.489435 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:16.489458 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:16.489467 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:16.489533 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:16.489678 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 07:29:16.489708 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:29:16.490947 21297 raft_consensus.cc:348] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51abb0833654130ba034add9a31da3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44773 } }
I20260926 07:29:16.491254 21297 raft_consensus.cc:374] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:16.491339 21297 raft_consensus.cc:729] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c51abb0833654130ba034add9a31da3c, State: Initialized, Role: FOLLOWER
I20260926 07:29:16.491863 21297 consensus_queue.cc:260] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51abb0833654130ba034add9a31da3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44773 } }
I20260926 07:29:16.492017 21297 raft_consensus.cc:388] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:16.492090 21297 raft_consensus.cc:482] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:16.492192 21297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:16.493139 21297 raft_consensus.cc:504] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51abb0833654130ba034add9a31da3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44773 } }
I20260926 07:29:16.493494 21297 leader_election.cc:302] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c51abb0833654130ba034add9a31da3c; no voters:
I20260926 07:29:16.493816 21297 leader_election.cc:288] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:16.493928 21307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:16.494163 21307 raft_consensus.cc:686] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [term 1 LEADER]: Becoming Leader. State: Replica: c51abb0833654130ba034add9a31da3c, State: Running, Role: LEADER
I20260926 07:29:16.494488 21307 consensus_queue.cc:237] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51abb0833654130ba034add9a31da3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44773 } }
I20260926 07:29:16.495198 21297 sys_catalog.cc:564] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:16.496273 21308 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c51abb0833654130ba034add9a31da3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51abb0833654130ba034add9a31da3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44773 } } }
I20260926 07:29:16.496411 21308 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:16.497454 21316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:16.496762 21309 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c51abb0833654130ba034add9a31da3c. Latest consensus state: current_term: 1 leader_uuid: "c51abb0833654130ba034add9a31da3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51abb0833654130ba034add9a31da3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44773 } } }
I20260926 07:29:16.498147 21309 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:16.500150 21316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:16.504982 21316 catalog_manager.cc:1347] Generated new cluster ID: 8af03371d47b430a99b23fa1a628a673
I20260926 07:29:16.505043 21316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:16.517844 21316 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:16.518198 21316 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:16.525089 21316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c51abb0833654130ba034add9a31da3c: Generated new TSK 0
I20260926 07:29:16.525899 21316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:16.538769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 21229
2026-09-26T07:29:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:16 dist-test-slave-g04g krb5kdc[21328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:16 dist-test-slave-g04g krb5kdc[21328](info): set up 2 sockets
Sep 26 07:29:16 dist-test-slave-g04g krb5kdc[21328](info): commencing operation
krb5kdc: starting...
W20260926 07:29:17.571419 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:17 dist-test-slave-g04g krb5kdc[21328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:29:17.693703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:17.801887 21341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:17.802139 21341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:17.802181 21341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:17.806028 21341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:17.806073 21341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:17.806092 21341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:17.806108 21341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:17.806123 21341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:17.811115 21341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:17.812004 21341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:17.812913 21341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:17.821048 21346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:17.821080 21347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:17.821080 21349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:17.821187 21341 server_base.cc:1034] running on GCE node
I20260926 07:29:17.821772 21341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:17.822839 21341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:17.824025 21341 hybrid_clock.cc:648] HybridClock initialized: now 1790407757824001 us; error 38 us; skew 500 ppm
Sep 26 07:29:17 dist-test-slave-g04g krb5kdc[21328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:17.826648 21341 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:29:17.827598 21341 webserver.cc:460] Webserver started at http://127.0.0.1:37711/ using document root <none> and password file <none>
I20260926 07:29:17.828140 21341 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:17.828226 21341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:17.828449 21341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:17.830389 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "9f9720c1cad548899e8e0b64f9f8d6f2"
format_stamp: "Formatted at 2026-09-26 07:29:17 on dist-test-slave-g04g"
server_key: "965470870c8826ab732e44226ada74e3"
server_key_iv: "375b7e33dbd249f35267f82f63938bb9"
server_key_version: "encryptionkey@0"
I20260926 07:29:17.830946 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f9720c1cad548899e8e0b64f9f8d6f2"
format_stamp: "Formatted at 2026-09-26 07:29:17 on dist-test-slave-g04g"
server_key: "965470870c8826ab732e44226ada74e3"
server_key_iv: "375b7e33dbd249f35267f82f63938bb9"
server_key_version: "encryptionkey@0"
I20260926 07:29:17.834371 21341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 07:29:17.836687 21356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:17.838045 21341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 07:29:17.838236 21341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "9f9720c1cad548899e8e0b64f9f8d6f2"
format_stamp: "Formatted at 2026-09-26 07:29:17 on dist-test-slave-g04g"
server_key: "965470870c8826ab732e44226ada74e3"
server_key_iv: "375b7e33dbd249f35267f82f63938bb9"
server_key_version: "encryptionkey@0"
I20260926 07:29:17.838367 21341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:17.857611 21341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:17.860929 21341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:17.861140 21341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:17.868667 21341 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37969
I20260926 07:29:17.868693 21408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37969 every 8 connection(s)
I20260926 07:29:17.869613 21341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:17.872478 21409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:17.877758 21409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2: Bootstrap starting.
I20260926 07:29:17.879827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 21341
I20260926 07:29:17.879962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:17.880264 392 external_mini_cluster.cc:1428] Setting key bc7e5aad26a20c8159046e0840f05ec9
I20260926 07:29:17.880335 21409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:17.881105 21409 log.cc:824] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:17.883311 21409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2: No bootstrap required, opened a new log
I20260926 07:29:17.887633 21409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9720c1cad548899e8e0b64f9f8d6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } }
I20260926 07:29:17.887908 21409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:17.887982 21409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9720c1cad548899e8e0b64f9f8d6f2, State: Initialized, Role: FOLLOWER
I20260926 07:29:17.888473 21409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9720c1cad548899e8e0b64f9f8d6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } }
I20260926 07:29:17.888594 21409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:17.888679 21409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:17.888784 21409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:17.889675 21409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9720c1cad548899e8e0b64f9f8d6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } }
Sep 26 07:29:17 dist-test-slave-g04g krb5kdc[21328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:29:17.890021 21409 leader_election.cc:302] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f9720c1cad548899e8e0b64f9f8d6f2; no voters:
I20260926 07:29:17.890322 21409 leader_election.cc:288] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:17.890451 21414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:17.890678 21414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [term 1 LEADER]: Becoming Leader. State: Replica: 9f9720c1cad548899e8e0b64f9f8d6f2, State: Running, Role: LEADER
I20260926 07:29:17.891089 21414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9720c1cad548899e8e0b64f9f8d6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } }
I20260926 07:29:17.891645 21409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:17.893548 21415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f9720c1cad548899e8e0b64f9f8d6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9720c1cad548899e8e0b64f9f8d6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } } }
I20260926 07:29:17.893663 21415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:17.894285 21416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f9720c1cad548899e8e0b64f9f8d6f2. Latest consensus state: current_term: 1 leader_uuid: "9f9720c1cad548899e8e0b64f9f8d6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9720c1cad548899e8e0b64f9f8d6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } } }
I20260926 07:29:17.894443 21416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:17.896433 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:17.881425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:33199 (local address 127.0.0.1:37969)
0926 07:29:17.881823 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:17.881862 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:17.881902 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:29:17.882739 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:17.882762 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:17.883198 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:17.883550 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:17.884320 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:17.885856 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:17.886766 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:17.887481 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:17.887672 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:17.890578 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:17.890586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:17.890603 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:17.890643 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:17.892826 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:17.893496 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:17.893504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:17.893509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:17.893653 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:17.894112 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:17.894118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:17.894120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:17.894676 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:17.894863 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:17.895231 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 07:29:17.895505 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":114,"threads_started":1}
W20260926 07:29:17.896798 21428 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:29:17.896888 21428 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:17.896991 21429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:17.897681 21429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:17.899621 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:17.903829 21429 catalog_manager.cc:1347] Generated new cluster ID: 40c5e7ac26a4429881f232dcad21934f
I20260926 07:29:17.903889 21429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:17.910143 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:17.901835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:45271 (local address 127.0.0.1:37969)
0926 07:29:17.901964 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:17.901968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:17.902079 (+ 111us) server_negotiation.cc:407] Connection header received
0926 07:29:17.902224 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:17.902227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:17.902278 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:17.902360 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:17.902976 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:17.903538 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:17.904653 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:17.905032 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:17.905146 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:17.906707 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:17.906711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:17.906714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:17.906741 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:17.908391 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:17.908890 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:17.908894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:17.908896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:17.908985 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:17.909299 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:17.909302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:17.909304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:17.909510 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:17.909670 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:17.909896 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 07:29:17.909991 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:29:17.936478 21429 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:17.937770 21429 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:17.953359 21429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f9720c1cad548899e8e0b64f9f8d6f2: Generated new TSK 0
I20260926 07:29:17.954233 21429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:17.967834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 21341
W20260926 07:29:17.973985 21326 connection.cc:535] client connection to 127.0.0.1:37969 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37969): Connection reset by peer (error 104)
2026-09-26T07:29:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:18 dist-test-slave-g04g krb5kdc[21440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:18 dist-test-slave-g04g krb5kdc[21440](info): set up 2 sockets
Sep 26 07:29:18 dist-test-slave-g04g krb5kdc[21440](info): commencing operation
krb5kdc: starting...
W20260926 07:29:19.027309 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:19 dist-test-slave-g04g krb5kdc[21440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:29:19.150328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41745
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:19.258620 21453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:19.258906 21453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:19.258981 21453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:19.262933 21453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:19.263024 21453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:19.263080 21453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:19.263142 21453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:19.263181 21453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:19.268384 21453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:19.269446 21453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:19.270469 21453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:19.278868 21458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:19.278868 21461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:19.278945 21459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:19.279145 21453 server_base.cc:1034] running on GCE node
I20260926 07:29:19.279736 21453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:19.280805 21453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:19.281966 21453 hybrid_clock.cc:648] HybridClock initialized: now 1790407759281953 us; error 25 us; skew 500 ppm
Sep 26 07:29:19 dist-test-slave-g04g krb5kdc[21440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:19.284684 21453 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:29:19.285523 21453 webserver.cc:460] Webserver started at http://127.0.0.1:36341/ using document root <none> and password file <none>
I20260926 07:29:19.286065 21453 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:19.286170 21453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:19.286393 21453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:19.288486 21453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "3df2b43764dc40838e793b4b897edbca"
format_stamp: "Formatted at 2026-09-26 07:29:19 on dist-test-slave-g04g"
server_key: "0ad5582780a3f074fe4d92b5e0567251"
server_key_iv: "2beaac8666778454950f80c038ef676b"
server_key_version: "encryptionkey@0"
I20260926 07:29:19.289039 21453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "3df2b43764dc40838e793b4b897edbca"
format_stamp: "Formatted at 2026-09-26 07:29:19 on dist-test-slave-g04g"
server_key: "0ad5582780a3f074fe4d92b5e0567251"
server_key_iv: "2beaac8666778454950f80c038ef676b"
server_key_version: "encryptionkey@0"
I20260926 07:29:19.292657 21453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:29:19.294973 21468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:19.296260 21453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:19.296396 21453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "3df2b43764dc40838e793b4b897edbca"
format_stamp: "Formatted at 2026-09-26 07:29:19 on dist-test-slave-g04g"
server_key: "0ad5582780a3f074fe4d92b5e0567251"
server_key_iv: "2beaac8666778454950f80c038ef676b"
server_key_version: "encryptionkey@0"
I20260926 07:29:19.296504 21453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:19.319352 21453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:19.322299 21453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:19.322518 21453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:19.329636 21453 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41745
I20260926 07:29:19.329664 21520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41745 every 8 connection(s)
I20260926 07:29:19.330695 21453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:19.333419 21521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:19.336158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 21453
I20260926 07:29:19.336311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:19.336647 392 external_mini_cluster.cc:1428] Setting key 20ff720daa89da5ed467b89fca7c587b
I20260926 07:29:19.341017 21521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca: Bootstrap starting.
I20260926 07:29:19.343683 21521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:19.344491 21521 log.cc:824] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:19.345917 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:19.338138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:52063 (local address 127.0.0.1:41745)
0926 07:29:19.338531 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:19.338548 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:19.338586 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:29:19.339348 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:19.339366 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:19.339708 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:19.340000 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:19.340306 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:19.341329 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:19.342525 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:19.342988 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:19.343201 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:19.344106 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:19.344129 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:19.344143 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:19.344514 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:19.344568 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:19.344589 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:19.344767 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:19.345016 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 07:29:19.345178 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":99,"threads_started":1}
I20260926 07:29:19.346535 21521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca: No bootstrap required, opened a new log
I20260926 07:29:19.348775 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:19.350726 21521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df2b43764dc40838e793b4b897edbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41745 } }
I20260926 07:29:19.351127 21521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:19.351289 21521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df2b43764dc40838e793b4b897edbca, State: Initialized, Role: FOLLOWER
I20260926 07:29:19.351785 21521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df2b43764dc40838e793b4b897edbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41745 } }
I20260926 07:29:19.351944 21521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:19.352030 21521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:19.352144 21521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:19.353078 21521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df2b43764dc40838e793b4b897edbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41745 } }
I20260926 07:29:19.353438 21521 leader_election.cc:302] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3df2b43764dc40838e793b4b897edbca; no voters:
I20260926 07:29:19.353775 21521 leader_election.cc:288] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:19.353842 21531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:19.354094 21531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [term 1 LEADER]: Becoming Leader. State: Replica: 3df2b43764dc40838e793b4b897edbca, State: Running, Role: LEADER
I20260926 07:29:19.354429 21531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df2b43764dc40838e793b4b897edbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41745 } }
I20260926 07:29:19.355309 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:19.351148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:46979 (local address 127.0.0.1:41745)
0926 07:29:19.351301 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:19.351307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:19.351498 (+ 191us) server_negotiation.cc:407] Connection header received
0926 07:29:19.351625 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:19.351628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:19.351683 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:19.351772 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:19.352042 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:19.352553 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:19.353446 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:19.353668 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:19.353832 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:19.354775 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:19.354778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:19.354780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:19.354791 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:19.354814 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:19.354824 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:19.354922 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:19.355061 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 07:29:19.355092 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:29:19.355211 21521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:19.355758 21531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3df2b43764dc40838e793b4b897edbca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df2b43764dc40838e793b4b897edbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41745 } } }
I20260926 07:29:19.355901 21531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:19.356289 21535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:19.356238 21532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3df2b43764dc40838e793b4b897edbca. Latest consensus state: current_term: 1 leader_uuid: "3df2b43764dc40838e793b4b897edbca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df2b43764dc40838e793b4b897edbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41745 } } }
I20260926 07:29:19.356412 21532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:19.359486 21535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:19.364837 21535 catalog_manager.cc:1347] Generated new cluster ID: 2cdf19e4cbd840b09278cf4d91e46167
I20260926 07:29:19.364902 21535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:19.374560 21535 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:19.375818 21535 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:19.398157 21535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3df2b43764dc40838e793b4b897edbca: Generated new TSK 0
I20260926 07:29:19.398977 21535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:19.404767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 21453
W20260926 07:29:19.410179 21438 connection.cc:535] client connection to 127.0.0.1:41745 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41745): Connection reset by peer (error 104)
2026-09-26T07:29:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:19 dist-test-slave-g04g krb5kdc[21552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:19 dist-test-slave-g04g krb5kdc[21552](info): set up 2 sockets
Sep 26 07:29:19 dist-test-slave-g04g krb5kdc[21552](info): commencing operation
krb5kdc: starting...
W20260926 07:29:20.443442 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:20 dist-test-slave-g04g krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:29:20.568491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39737
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:20.697094 21565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:20.697515 21565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:20.697675 21565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:20.706002 21565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:20.706115 21565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:20.706161 21565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:20.706208 21565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:20.706252 21565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:20.716820 21565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:20.717916 21565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:20.718946 21565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:20.727453 21571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:20.727449 21573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:20.727665 21565 server_base.cc:1034] running on GCE node
W20260926 07:29:20.727519 21570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:20.728305 21565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:20.729432 21565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:20.730605 21565 hybrid_clock.cc:648] HybridClock initialized: now 1790407760730584 us; error 29 us; skew 500 ppm
Sep 26 07:29:20 dist-test-slave-g04g krb5kdc[21552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:20.733541 21565 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:29:20.734411 21565 webserver.cc:460] Webserver started at http://127.0.0.1:38145/ using document root <none> and password file <none>
I20260926 07:29:20.734916 21565 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:20.734978 21565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:20.735224 21565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:20.737195 21565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "6c1716c730c54210ae91f184e5472b8e"
format_stamp: "Formatted at 2026-09-26 07:29:20 on dist-test-slave-g04g"
server_key: "58c25a4463218406562a2c52a4f93256"
server_key_iv: "39e685c66ce5ed9caf8796b4479a179c"
server_key_version: "encryptionkey@0"
I20260926 07:29:20.737725 21565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c1716c730c54210ae91f184e5472b8e"
format_stamp: "Formatted at 2026-09-26 07:29:20 on dist-test-slave-g04g"
server_key: "58c25a4463218406562a2c52a4f93256"
server_key_iv: "39e685c66ce5ed9caf8796b4479a179c"
server_key_version: "encryptionkey@0"
I20260926 07:29:20.741465 21565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260926 07:29:20.743782 21580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:20.745077 21565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:20.745209 21565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "6c1716c730c54210ae91f184e5472b8e"
format_stamp: "Formatted at 2026-09-26 07:29:20 on dist-test-slave-g04g"
server_key: "58c25a4463218406562a2c52a4f93256"
server_key_iv: "39e685c66ce5ed9caf8796b4479a179c"
server_key_version: "encryptionkey@0"
I20260926 07:29:20.745340 21565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:20.749671 21565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:20.752755 21565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:20.752942 21565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:20.760177 21565 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39737
I20260926 07:29:20.760205 21632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39737 every 8 connection(s)
I20260926 07:29:20.761039 21565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:20.763711 21633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:20.764500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 21565
I20260926 07:29:20.764627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:20.764962 392 external_mini_cluster.cc:1428] Setting key 72e8706e490bae2c7c0006788ed3187c
I20260926 07:29:20.769783 21633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e: Bootstrap starting.
I20260926 07:29:20.769824 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:20.766289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:34895 (local address 127.0.0.1:39737)
0926 07:29:20.766690 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:20.766704 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:20.766736 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:29:20.767413 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:20.767430 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:20.767703 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:20.768028 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:20.768737 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:20.768742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:20.768753 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:20.768763 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:20.768810 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:20.768830 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:20.768970 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:20.769038 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 07:29:20.769187 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260926 07:29:20.771973 21633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:20.772442 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:20.772751 21633 log.cc:824] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:20.774648 21633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e: No bootstrap required, opened a new log
I20260926 07:29:20.775835 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:20.774400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:59591 (local address 127.0.0.1:39737)
0926 07:29:20.774510 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:20.774513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:20.774614 (+ 101us) server_negotiation.cc:407] Connection header received
0926 07:29:20.774747 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:20.774749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:20.774787 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:20.774850 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:20.775481 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:20.775484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:20.775485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:20.775494 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:20.775516 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:20.775525 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:20.775589 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:20.775688 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 07:29:20.775714 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:29:20.778287 21633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1716c730c54210ae91f184e5472b8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39737 } }
I20260926 07:29:20.778494 21633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:20.778527 21633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1716c730c54210ae91f184e5472b8e, State: Initialized, Role: FOLLOWER
I20260926 07:29:20.778853 21633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1716c730c54210ae91f184e5472b8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39737 } }
I20260926 07:29:20.778957 21633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:20.779002 21633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:20.779124 21633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:20.779893 21633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1716c730c54210ae91f184e5472b8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39737 } }
I20260926 07:29:20.780215 21633 leader_election.cc:302] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c1716c730c54210ae91f184e5472b8e; no voters:
I20260926 07:29:20.780529 21633 leader_election.cc:288] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:20.780593 21643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:20.780786 21643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [term 1 LEADER]: Becoming Leader. State: Replica: 6c1716c730c54210ae91f184e5472b8e, State: Running, Role: LEADER
I20260926 07:29:20.781104 21643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1716c730c54210ae91f184e5472b8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39737 } }
I20260926 07:29:20.781757 21633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:20.782974 21645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c1716c730c54210ae91f184e5472b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1716c730c54210ae91f184e5472b8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39737 } } }
I20260926 07:29:20.783016 21644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c1716c730c54210ae91f184e5472b8e. Latest consensus state: current_term: 1 leader_uuid: "6c1716c730c54210ae91f184e5472b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1716c730c54210ae91f184e5472b8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39737 } } }
I20260926 07:29:20.783187 21644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:20.783442 21645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:20.783885 21652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:20.786765 21652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:20.791594 21652 catalog_manager.cc:1347] Generated new cluster ID: f7304016ceb043d8840dbbfeda8c8c6a
I20260926 07:29:20.791661 21652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:20.846180 21652 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:20.846529 21652 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:20.856076 21652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c1716c730c54210ae91f184e5472b8e: Generated new TSK 0
I20260926 07:29:20.856756 21652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:20.869027 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 21565
2026-09-26T07:29:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1464 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:20 dist-test-slave-g04g krb5kdc[21664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:20 dist-test-slave-g04g krb5kdc[21664](info): set up 2 sockets
Sep 26 07:29:20 dist-test-slave-g04g krb5kdc[21664](info): commencing operation
krb5kdc: starting...
W20260926 07:29:21.907346 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:21 dist-test-slave-g04g krb5kdc[21664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:29:22.027019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37673
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:22.133817 21677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:22.134052 21677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:22.134091 21677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:22.138027 21677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:22.138079 21677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:22.138098 21677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:22.138159 21677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:22.138195 21677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:22.143293 21677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37673
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:22.144244 21677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:22.145200 21677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:22.153398 21685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:22.153414 21682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:22.153438 21683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:22.154143 21677 server_base.cc:1034] running on GCE node
I20260926 07:29:22.154724 21677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:22.155859 21677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:22.157033 21677 hybrid_clock.cc:648] HybridClock initialized: now 1790407762157006 us; error 40 us; skew 500 ppm
Sep 26 07:29:22 dist-test-slave-g04g krb5kdc[21664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:22.159984 21677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:29:22.160925 21677 webserver.cc:460] Webserver started at http://127.0.0.1:34065/ using document root <none> and password file <none>
I20260926 07:29:22.161473 21677 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:22.161557 21677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:22.161765 21677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:22.163856 21677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "d40ac3eea5c44acca65e02ba3a535967"
format_stamp: "Formatted at 2026-09-26 07:29:22 on dist-test-slave-g04g"
server_key: "eb26b56fe3d56587819df18ed57366f9"
server_key_iv: "838766567da8d9b91123cafbd1d2a725"
server_key_version: "encryptionkey@0"
I20260926 07:29:22.164409 21677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "d40ac3eea5c44acca65e02ba3a535967"
format_stamp: "Formatted at 2026-09-26 07:29:22 on dist-test-slave-g04g"
server_key: "eb26b56fe3d56587819df18ed57366f9"
server_key_iv: "838766567da8d9b91123cafbd1d2a725"
server_key_version: "encryptionkey@0"
I20260926 07:29:22.167934 21677 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 07:29:22.170189 21692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:22.171540 21677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:22.171686 21677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "d40ac3eea5c44acca65e02ba3a535967"
format_stamp: "Formatted at 2026-09-26 07:29:22 on dist-test-slave-g04g"
server_key: "eb26b56fe3d56587819df18ed57366f9"
server_key_iv: "838766567da8d9b91123cafbd1d2a725"
server_key_version: "encryptionkey@0"
I20260926 07:29:22.171806 21677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:22.194119 21677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:22.197116 21677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:22.197328 21677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:22.204540 21677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37673
I20260926 07:29:22.204571 21744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37673 every 8 connection(s)
I20260926 07:29:22.205474 21677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:22.208117 21745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:22.213267 21745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967: Bootstrap starting.
I20260926 07:29:22.213280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 21677
I20260926 07:29:22.213460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:22.213757 392 external_mini_cluster.cc:1428] Setting key c10c9f45c9ff4fadabb7dba4ff594cd3
I20260926 07:29:22.215916 21745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:22.216830 21745 log.cc:824] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:22.218876 21745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967: No bootstrap required, opened a new log
I20260926 07:29:22.222709 21745 raft_consensus.cc:348] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40ac3eea5c44acca65e02ba3a535967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37673 } }
I20260926 07:29:22.222966 21745 raft_consensus.cc:374] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:22.223050 21745 raft_consensus.cc:729] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40ac3eea5c44acca65e02ba3a535967, State: Initialized, Role: FOLLOWER
Sep 26 07:29:22 dist-test-slave-g04g krb5kdc[21664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:29:22.223582 21745 consensus_queue.cc:260] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40ac3eea5c44acca65e02ba3a535967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37673 } }
I20260926 07:29:22.223749 21745 raft_consensus.cc:388] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:22.223852 21745 raft_consensus.cc:482] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:22.223971 21745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:22.224784 21745 raft_consensus.cc:504] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40ac3eea5c44acca65e02ba3a535967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37673 } }
I20260926 07:29:22.225103 21745 leader_election.cc:302] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d40ac3eea5c44acca65e02ba3a535967; no voters:
I20260926 07:29:22.225421 21745 leader_election.cc:288] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:22.225485 21750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:22.225704 21750 raft_consensus.cc:686] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [term 1 LEADER]: Becoming Leader. State: Replica: d40ac3eea5c44acca65e02ba3a535967, State: Running, Role: LEADER
I20260926 07:29:22.226042 21750 consensus_queue.cc:237] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40ac3eea5c44acca65e02ba3a535967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37673 } }
I20260926 07:29:22.226653 21745 sys_catalog.cc:564] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:22.227720 21751 sys_catalog.cc:455] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d40ac3eea5c44acca65e02ba3a535967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40ac3eea5c44acca65e02ba3a535967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37673 } } }
I20260926 07:29:22.228086 21751 sys_catalog.cc:458] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:22.227973 21752 sys_catalog.cc:455] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d40ac3eea5c44acca65e02ba3a535967. Latest consensus state: current_term: 1 leader_uuid: "d40ac3eea5c44acca65e02ba3a535967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40ac3eea5c44acca65e02ba3a535967" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37673 } } }
I20260926 07:29:22.228411 21752 sys_catalog.cc:458] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:22.229853 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:22.215165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:48289 (local address 127.0.0.1:37673)
0926 07:29:22.215557 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:22.215575 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:22.215615 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:29:22.216481 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:22.216503 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:22.216934 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:22.217283 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:22.218063 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:22.219491 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:22.220309 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:22.221001 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:22.221161 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:22.223890 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:22.223895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:22.223927 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:22.223960 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:22.225918 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:22.226481 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:22.226487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:22.226492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:22.226621 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:22.226960 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:22.226964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:22.226966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:22.227573 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:22.227795 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:22.228712 (+ 917us) server_negotiation.cc:299] Negotiation successful
0926 07:29:22.228962 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":117,"threads_started":1}
W20260926 07:29:22.231838 21766 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:29:22.231925 21766 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:22.232036 21760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:22.232923 21760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:22.233069 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:22.238283 21760 catalog_manager.cc:1347] Generated new cluster ID: 3a82d72d854046d7a1ec6650a2f0590e
I20260926 07:29:22.238338 21760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:22.243707 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:22.235252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:58479 (local address 127.0.0.1:37673)
0926 07:29:22.235349 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:22.235353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:22.235675 (+ 322us) server_negotiation.cc:407] Connection header received
0926 07:29:22.235783 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:22.235786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:22.235831 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:22.235914 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:22.236697 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:22.237258 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:22.238130 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:22.238465 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:22.238593 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:22.240249 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:22.240252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:22.240255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:22.240280 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:22.241903 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:22.242426 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:22.242429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:22.242431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:22.242506 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:22.242839 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:22.242842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:22.242844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:22.243022 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:22.243205 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:22.243466 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 07:29:22.243563 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 07:29:22.247129 21760 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:22.247965 21760 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:22.254447 21760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d40ac3eea5c44acca65e02ba3a535967: Generated new TSK 0
I20260926 07:29:22.255146 21760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:22.260087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 21677
W20260926 07:29:22.264931 21662 connection.cc:535] client connection to 127.0.0.1:37673 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37673): Connection reset by peer (error 104)
2026-09-26T07:29:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:22 dist-test-slave-g04g krb5kdc[21776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:22 dist-test-slave-g04g krb5kdc[21776](info): set up 2 sockets
Sep 26 07:29:22 dist-test-slave-g04g krb5kdc[21776](info): commencing operation
krb5kdc: starting...
W20260926 07:29:23.311331 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:23 dist-test-slave-g04g krb5kdc[21776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:29:23.433967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:23.544940 21789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:23.545241 21789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:23.545285 21789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:23.549293 21789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:23.549353 21789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:23.549373 21789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:23.549391 21789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:23.549407 21789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:23.554692 21789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38079
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:23.555753 21789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:23.556780 21789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:23.565894 21795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:23.565883 21794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:23.566035 21797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:23.566293 21789 server_base.cc:1034] running on GCE node
I20260926 07:29:23.566865 21789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:23.568032 21789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:23.569226 21789 hybrid_clock.cc:648] HybridClock initialized: now 1790407763569197 us; error 53 us; skew 500 ppm
Sep 26 07:29:23 dist-test-slave-g04g krb5kdc[21776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:23.571947 21789 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:29:23.572867 21789 webserver.cc:460] Webserver started at http://127.0.0.1:37979/ using document root <none> and password file <none>
I20260926 07:29:23.573434 21789 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:23.573524 21789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:23.573764 21789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:23.575927 21789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "2a13539c37df47b683a1ed865e4ed28a"
format_stamp: "Formatted at 2026-09-26 07:29:23 on dist-test-slave-g04g"
server_key: "612481a04e143a5586bebd08548fffb0"
server_key_iv: "85aa4e6b3e58a4d52df235b7943ab778"
server_key_version: "encryptionkey@0"
I20260926 07:29:23.576488 21789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a13539c37df47b683a1ed865e4ed28a"
format_stamp: "Formatted at 2026-09-26 07:29:23 on dist-test-slave-g04g"
server_key: "612481a04e143a5586bebd08548fffb0"
server_key_iv: "85aa4e6b3e58a4d52df235b7943ab778"
server_key_version: "encryptionkey@0"
I20260926 07:29:23.580178 21789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 07:29:23.583294 21804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:23.584559 21789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:29:23.584714 21789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "2a13539c37df47b683a1ed865e4ed28a"
format_stamp: "Formatted at 2026-09-26 07:29:23 on dist-test-slave-g04g"
server_key: "612481a04e143a5586bebd08548fffb0"
server_key_iv: "85aa4e6b3e58a4d52df235b7943ab778"
server_key_version: "encryptionkey@0"
I20260926 07:29:23.584838 21789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:23.600309 21789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:23.603178 21789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:23.603376 21789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:23.610459 21789 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38079
I20260926 07:29:23.610478 21856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38079 every 8 connection(s)
I20260926 07:29:23.611459 21789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:23.614054 21857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:23.619424 21857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a: Bootstrap starting.
I20260926 07:29:23.620098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 21789
I20260926 07:29:23.620234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:23.620502 392 external_mini_cluster.cc:1428] Setting key 4b0eab8a643e107fac9497227ea5d59a
I20260926 07:29:23.622402 21857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:23.623202 21857 log.cc:824] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:23.625006 21857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a: No bootstrap required, opened a new log
I20260926 07:29:23.628419 21857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a13539c37df47b683a1ed865e4ed28a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38079 } }
I20260926 07:29:23.628638 21857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:23.628713 21857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a13539c37df47b683a1ed865e4ed28a, State: Initialized, Role: FOLLOWER
I20260926 07:29:23.629195 21857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a13539c37df47b683a1ed865e4ed28a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38079 } }
I20260926 07:29:23.629354 21857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:23.629446 21857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:23.629566 21857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:23.630533 21857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a13539c37df47b683a1ed865e4ed28a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38079 } }
I20260926 07:29:23.631018 21857 leader_election.cc:302] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a13539c37df47b683a1ed865e4ed28a; no voters:
I20260926 07:29:23.631485 21857 leader_election.cc:288] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:23.631588 21862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:23.631855 21862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [term 1 LEADER]: Becoming Leader. State: Replica: 2a13539c37df47b683a1ed865e4ed28a, State: Running, Role: LEADER
I20260926 07:29:23.632222 21862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a13539c37df47b683a1ed865e4ed28a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38079 } }
I20260926 07:29:23.633046 21857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:23.634166 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:23.621878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:40789 (local address 127.0.0.1:38079)
0926 07:29:23.622318 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:23.622333 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:23.622368 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:29:23.623164 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:23.623183 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:23.623583 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:23.623929 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:23.624349 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:23.625382 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:23.626305 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:23.626800 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:23.626981 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:23.631687 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:23.631694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:23.631712 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:23.632279 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:23.632361 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:23.632394 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:23.632621 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:23.632958 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 07:29:23.633117 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":129,"threads_started":1}
I20260926 07:29:23.634132 21864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a13539c37df47b683a1ed865e4ed28a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a13539c37df47b683a1ed865e4ed28a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38079 } } }
I20260926 07:29:23.634352 21863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a13539c37df47b683a1ed865e4ed28a. Latest consensus state: current_term: 1 leader_uuid: "2a13539c37df47b683a1ed865e4ed28a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a13539c37df47b683a1ed865e4ed28a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38079 } } }
I20260926 07:29:23.634478 21863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:23.634478 21864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:23.637800 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 07:29:23.638787 21878 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:29:23.638867 21878 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:23.638927 21871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:23.639889 21871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:23.645094 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:23.640117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:43749 (local address 127.0.0.1:38079)
0926 07:29:23.640249 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:23.640254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:23.640552 (+ 298us) server_negotiation.cc:407] Connection header received
0926 07:29:23.640680 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:23.640683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:23.640730 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:23.640811 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:23.641156 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:23.641941 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:23.642781 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:23.643027 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:23.643166 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:23.644129 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:23.644133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:23.644136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:23.644151 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:23.644180 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:23.644195 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:23.644295 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:23.644833 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 07:29:23.644881 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:29:23.646381 21871 catalog_manager.cc:1347] Generated new cluster ID: c63244c1dd3940daa1bda7ef541fd927
I20260926 07:29:23.646443 21871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:23.652882 21871 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:23.653841 21871 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:23.666512 21871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a13539c37df47b683a1ed865e4ed28a: Generated new TSK 0
I20260926 07:29:23.667320 21871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:23.673213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 21789
W20260926 07:29:23.679512 21774 connection.cc:535] client connection to 127.0.0.1:38079 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38079): Connection reset by peer (error 104)
2026-09-26T07:29:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:23 dist-test-slave-g04g krb5kdc[21888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:23 dist-test-slave-g04g krb5kdc[21888](info): set up 2 sockets
Sep 26 07:29:23 dist-test-slave-g04g krb5kdc[21888](info): commencing operation
krb5kdc: starting...
W20260926 07:29:24.775454 392 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:24 dist-test-slave-g04g krb5kdc[21888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:29:24.893668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:25.006559 21901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:25.006860 21901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:25.006940 21901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:25.010957 21901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:25.011088 21901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:25.011180 21901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:25.011226 21901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:25.011265 21901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:25.016592 21901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:25.017670 21901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:25.018710 21901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:25.027024 21909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:25.027083 21906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:25.027058 21907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:25.027917 21901 server_base.cc:1034] running on GCE node
I20260926 07:29:25.028693 21901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:25.030031 21901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:25.031211 21901 hybrid_clock.cc:648] HybridClock initialized: now 1790407765031189 us; error 46 us; skew 500 ppm
Sep 26 07:29:25 dist-test-slave-g04g krb5kdc[21888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:25.033759 21901 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:29:25.034600 21901 webserver.cc:460] Webserver started at http://127.0.0.1:43811/ using document root <none> and password file <none>
I20260926 07:29:25.035219 21901 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:25.035308 21901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:25.035530 21901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:25.037504 21901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "c80f2818b976444a864cce402f7b5b89"
format_stamp: "Formatted at 2026-09-26 07:29:25 on dist-test-slave-g04g"
server_key: "4e1c1a2bd1d3d0b1d19e28cc42112ff0"
server_key_iv: "67574f5cc943a0ca01b97d235726c8ab"
server_key_version: "encryptionkey@0"
I20260926 07:29:25.038054 21901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "c80f2818b976444a864cce402f7b5b89"
format_stamp: "Formatted at 2026-09-26 07:29:25 on dist-test-slave-g04g"
server_key: "4e1c1a2bd1d3d0b1d19e28cc42112ff0"
server_key_iv: "67574f5cc943a0ca01b97d235726c8ab"
server_key_version: "encryptionkey@0"
I20260926 07:29:25.041587 21901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260926 07:29:25.043949 21916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:25.045171 21901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 07:29:25.045305 21901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "c80f2818b976444a864cce402f7b5b89"
format_stamp: "Formatted at 2026-09-26 07:29:25 on dist-test-slave-g04g"
server_key: "4e1c1a2bd1d3d0b1d19e28cc42112ff0"
server_key_iv: "67574f5cc943a0ca01b97d235726c8ab"
server_key_version: "encryptionkey@0"
I20260926 07:29:25.045418 21901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:25.049804 21901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:25.052541 21901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:25.052726 21901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:25.059231 21901 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34121
I20260926 07:29:25.059293 21968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34121 every 8 connection(s)
I20260926 07:29:25.060181 21901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:25.062762 21969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:25.068214 21969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89: Bootstrap starting.
I20260926 07:29:25.069008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 21901
I20260926 07:29:25.069139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:25.069446 392 external_mini_cluster.cc:1428] Setting key 64363001fbf9fa9bfbb402e6683b05da
I20260926 07:29:25.071148 21969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:25.071866 21969 log.cc:824] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:25.074031 21969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89: No bootstrap required, opened a new log
I20260926 07:29:25.074535 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:25.070583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:44965 (local address 127.0.0.1:34121)
0926 07:29:25.070980 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:25.070993 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:25.071067 (+ 74us) server_negotiation.cc:407] Connection header received
0926 07:29:25.071749 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:25.071765 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:25.072105 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:25.072388 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:25.073198 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:25.073203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:25.073217 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:25.073228 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:25.073290 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:25.073318 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:25.073436 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:25.073508 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 07:29:25.073653 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":121,"threads_started":1}
I20260926 07:29:25.077224 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:25.078094 21969 raft_consensus.cc:348] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80f2818b976444a864cce402f7b5b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34121 } }
I20260926 07:29:25.078320 21969 raft_consensus.cc:374] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:25.078380 21969 raft_consensus.cc:729] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80f2818b976444a864cce402f7b5b89, State: Initialized, Role: FOLLOWER
I20260926 07:29:25.078795 21969 consensus_queue.cc:260] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80f2818b976444a864cce402f7b5b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34121 } }
I20260926 07:29:25.078950 21969 raft_consensus.cc:388] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:25.079030 21969 raft_consensus.cc:482] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:25.079176 21969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:25.080384 21969 raft_consensus.cc:504] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80f2818b976444a864cce402f7b5b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34121 } }
I20260926 07:29:25.080780 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:25.079179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:35547 (local address 127.0.0.1:34121)
0926 07:29:25.079299 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:25.079303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:25.079510 (+ 207us) server_negotiation.cc:407] Connection header received
0926 07:29:25.079589 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:25.079592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:25.079624 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:25.079696 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:25.080386 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:25.080389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:25.080394 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:25.080408 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:25.080435 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:25.080448 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:25.080538 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:25.080633 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 07:29:25.080669 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:29:25.080747 21969 leader_election.cc:302] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c80f2818b976444a864cce402f7b5b89; no voters:
I20260926 07:29:25.081058 21969 leader_election.cc:288] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:25.081166 21979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:25.081370 21979 raft_consensus.cc:686] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [term 1 LEADER]: Becoming Leader. State: Replica: c80f2818b976444a864cce402f7b5b89, State: Running, Role: LEADER
I20260926 07:29:25.081674 21979 consensus_queue.cc:237] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80f2818b976444a864cce402f7b5b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34121 } }
I20260926 07:29:25.082226 21969 sys_catalog.cc:564] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:25.082701 21979 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c80f2818b976444a864cce402f7b5b89. Latest consensus state: current_term: 1 leader_uuid: "c80f2818b976444a864cce402f7b5b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80f2818b976444a864cce402f7b5b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34121 } } }
I20260926 07:29:25.082816 21979 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:25.083221 21984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:25.083308 21980 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c80f2818b976444a864cce402f7b5b89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80f2818b976444a864cce402f7b5b89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34121 } } }
I20260926 07:29:25.083468 21980 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:25.086325 21984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:25.091516 21984 catalog_manager.cc:1347] Generated new cluster ID: 6e15d5489d6346dbb982d485f49ecb81
I20260926 07:29:25.091583 21984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:25.103672 21984 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:25.104022 21984 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:25.108608 21984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c80f2818b976444a864cce402f7b5b89: Generated new TSK 0
I20260926 07:29:25.109306 21984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:25.119253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 21901
2026-09-26T07:29:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:25 dist-test-slave-g04g krb5kdc[22000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:25 dist-test-slave-g04g krb5kdc[22000](info): set up 2 sockets
Sep 26 07:29:25 dist-test-slave-g04g krb5kdc[22000](info): commencing operation
krb5kdc: starting...
W20260926 07:29:26.155377 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:26 dist-test-slave-g04g krb5kdc[22000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:29:26.274662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38261
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:26.380679 22013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:26.380914 22013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:26.380956 22013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:26.384814 22013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:26.384863 22013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:26.384882 22013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:26.384898 22013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:26.384913 22013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:26.390096 22013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:26.390972 22013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:26.391957 22013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:26.400224 22018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:26.400215 22019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:26.400277 22021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:26.400578 22013 server_base.cc:1034] running on GCE node
I20260926 07:29:26.401088 22013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:26.402156 22013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:26.403337 22013 hybrid_clock.cc:648] HybridClock initialized: now 1790407766403314 us; error 34 us; skew 500 ppm
Sep 26 07:29:26 dist-test-slave-g04g krb5kdc[22000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:26.405987 22013 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:29:26.406785 22013 webserver.cc:460] Webserver started at http://127.0.98.62:35149/ using document root <none> and password file <none>
I20260926 07:29:26.407379 22013 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:26.407469 22013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:26.407730 22013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:26.409750 22013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "0b963229b9984f3b975143787437dc3d"
format_stamp: "Formatted at 2026-09-26 07:29:26 on dist-test-slave-g04g"
server_key: "e49dfb1baf246a218b2b63ac19b9f97b"
server_key_iv: "1822340cb15f0b8b8b075a47a569fabf"
server_key_version: "encryptionkey@0"
I20260926 07:29:26.410290 22013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b963229b9984f3b975143787437dc3d"
format_stamp: "Formatted at 2026-09-26 07:29:26 on dist-test-slave-g04g"
server_key: "e49dfb1baf246a218b2b63ac19b9f97b"
server_key_iv: "1822340cb15f0b8b8b075a47a569fabf"
server_key_version: "encryptionkey@0"
I20260926 07:29:26.413749 22013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:29:26.416039 22028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:26.417119 22013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:26.417251 22013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "0b963229b9984f3b975143787437dc3d"
format_stamp: "Formatted at 2026-09-26 07:29:26 on dist-test-slave-g04g"
server_key: "e49dfb1baf246a218b2b63ac19b9f97b"
server_key_iv: "1822340cb15f0b8b8b075a47a569fabf"
server_key_version: "encryptionkey@0"
I20260926 07:29:26.417363 22013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:26.439100 22013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:26.442433 22013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:26.442638 22013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:26.449569 22013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38261
I20260926 07:29:26.449680 22080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38261 every 8 connection(s)
I20260926 07:29:26.450577 22013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:26.453591 22081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:26.459384 22081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d: Bootstrap starting.
I20260926 07:29:26.460184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 22013
I20260926 07:29:26.460330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:26.460600 392 external_mini_cluster.cc:1428] Setting key ceb7d131850e400ba10149863393d351
I20260926 07:29:26.462111 22081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:26.462821 22081 log.cc:824] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:26.465268 22081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d: No bootstrap required, opened a new log
I20260926 07:29:26.469165 22081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b963229b9984f3b975143787437dc3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38261 } }
I20260926 07:29:26.469367 22081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:26.469426 22081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b963229b9984f3b975143787437dc3d, State: Initialized, Role: FOLLOWER
Sep 26 07:29:26 dist-test-slave-g04g krb5kdc[22000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:26.469900 22081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b963229b9984f3b975143787437dc3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38261 } }
I20260926 07:29:26.470019 22081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:26.470100 22081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:26.470216 22081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:26.471462 22081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b963229b9984f3b975143787437dc3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38261 } }
I20260926 07:29:26.471848 22081 leader_election.cc:302] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b963229b9984f3b975143787437dc3d; no voters:
I20260926 07:29:26.472198 22081 leader_election.cc:288] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:26.472401 22086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:26.472796 22086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [term 1 LEADER]: Becoming Leader. State: Replica: 0b963229b9984f3b975143787437dc3d, State: Running, Role: LEADER
I20260926 07:29:26.473142 22086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b963229b9984f3b975143787437dc3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38261 } }
I20260926 07:29:26.473600 22081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:26.474581 22086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b963229b9984f3b975143787437dc3d. Latest consensus state: current_term: 1 leader_uuid: "0b963229b9984f3b975143787437dc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b963229b9984f3b975143787437dc3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38261 } } }
I20260926 07:29:26.475517 22086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:26.475379 22087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b963229b9984f3b975143787437dc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b963229b9984f3b975143787437dc3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38261 } } }
I20260926 07:29:26.475965 22087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:26.477180 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:26.461718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34539 (local address 127.0.98.62:38261)
0926 07:29:26.462181 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:26.462201 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:26.462250 (+ 49us) server_negotiation.cc:407] Connection header received
0926 07:29:26.462865 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:26.462906 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:26.463315 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:26.463598 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:26.464316 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:26.465750 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:26.466647 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:26.467387 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:26.467549 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:26.470318 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:26.470325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:26.470341 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:26.470373 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:26.472472 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:26.473125 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:26.473130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:26.473136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:26.473295 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:26.473809 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:26.473813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:26.473815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:26.474409 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:26.475676 (+ 1267us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:26.475905 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 07:29:26.476160 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":153,"threads_started":1}
W20260926 07:29:26.478236 22100 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:29:26.478323 22100 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:26.478384 22101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:26.479176 22101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:26.480584 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:26.484923 22101 catalog_manager.cc:1347] Generated new cluster ID: 9e96148c177f4d82aafcfc73e105e60b
I20260926 07:29:26.484978 22101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:26.491552 22101 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:26.492615 22101 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:26.493937 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:26.482566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49833 (local address 127.0.98.62:38261)
0926 07:29:26.482672 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:26.482676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:26.483284 (+ 608us) server_negotiation.cc:407] Connection header received
0926 07:29:26.483316 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:26.483318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:26.483352 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:26.483442 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:26.484265 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:26.485565 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:26.486933 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:26.487353 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:26.487448 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:26.489786 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:26.489792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:26.489795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:26.489828 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:26.491758 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:26.492466 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:26.492470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:26.492472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:26.492558 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:26.492981 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:26.492984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:26.492986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:26.493181 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:26.493314 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:26.493686 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 07:29:26.493781 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":41}
I20260926 07:29:26.500818 22101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b963229b9984f3b975143787437dc3d: Generated new TSK 0
I20260926 07:29:26.501621 22101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:26.506709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 22013
W20260926 07:29:26.512048 21998 connection.cc:535] client connection to 127.0.98.62:38261 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38261): Connection reset by peer (error 104)
2026-09-26T07:29:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:26 dist-test-slave-g04g krb5kdc[22112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:26 dist-test-slave-g04g krb5kdc[22112](info): set up 2 sockets
Sep 26 07:29:26 dist-test-slave-g04g krb5kdc[22112](info): commencing operation
krb5kdc: starting...
W20260926 07:29:27.531282 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:27 dist-test-slave-g04g krb5kdc[22112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:29:27.651185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46237
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:27.759192 22125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:27.759451 22125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:27.759526 22125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:27.763613 22125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:27.763674 22125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:27.763693 22125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:27.763756 22125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:27.763782 22125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:27.769172 22125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:27.770243 22125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:27.771298 22125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:27.779918 22133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:27.779927 22131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:27.779927 22130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:27.780306 22125 server_base.cc:1034] running on GCE node
I20260926 07:29:27.780836 22125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:27.781942 22125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:27.783106 22125 hybrid_clock.cc:648] HybridClock initialized: now 1790407767783088 us; error 29 us; skew 500 ppm
Sep 26 07:29:27 dist-test-slave-g04g krb5kdc[22112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:27.785836 22125 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:29:27.786639 22125 webserver.cc:460] Webserver started at http://127.0.98.62:44637/ using document root <none> and password file <none>
I20260926 07:29:27.787221 22125 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:27.787287 22125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:27.787451 22125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:27.789587 22125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "eab103b383a54be7878bbd4bdbcb37f5"
format_stamp: "Formatted at 2026-09-26 07:29:27 on dist-test-slave-g04g"
server_key: "20c2796a6f7a89ac49a701a2ae178cb4"
server_key_iv: "d5b1fa7864ce963f0a9beb452f091005"
server_key_version: "encryptionkey@0"
I20260926 07:29:27.790251 22125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "eab103b383a54be7878bbd4bdbcb37f5"
format_stamp: "Formatted at 2026-09-26 07:29:27 on dist-test-slave-g04g"
server_key: "20c2796a6f7a89ac49a701a2ae178cb4"
server_key_iv: "d5b1fa7864ce963f0a9beb452f091005"
server_key_version: "encryptionkey@0"
I20260926 07:29:27.793916 22125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:29:27.796430 22140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:27.797667 22125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:29:27.797794 22125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "eab103b383a54be7878bbd4bdbcb37f5"
format_stamp: "Formatted at 2026-09-26 07:29:27 on dist-test-slave-g04g"
server_key: "20c2796a6f7a89ac49a701a2ae178cb4"
server_key_iv: "d5b1fa7864ce963f0a9beb452f091005"
server_key_version: "encryptionkey@0"
I20260926 07:29:27.797905 22125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:27.826736 22125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:27.830302 22125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:27.830535 22125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:27.837760 22125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46237
I20260926 07:29:27.837833 22192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46237 every 8 connection(s)
I20260926 07:29:27.838773 22125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:27.841588 22193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:27.847118 22193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5: Bootstrap starting.
I20260926 07:29:27.847471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 22125
I20260926 07:29:27.847608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:27.847882 392 external_mini_cluster.cc:1428] Setting key 0ae853404550a386638d2b88843da69e
I20260926 07:29:27.849773 22193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:27.850507 22193 log.cc:824] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:27.852332 22193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5: No bootstrap required, opened a new log
I20260926 07:29:27.855870 22193 raft_consensus.cc:348] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab103b383a54be7878bbd4bdbcb37f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46237 } }
I20260926 07:29:27.856171 22193 raft_consensus.cc:374] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:27.856249 22193 raft_consensus.cc:729] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab103b383a54be7878bbd4bdbcb37f5, State: Initialized, Role: FOLLOWER
I20260926 07:29:27.856724 22193 consensus_queue.cc:260] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab103b383a54be7878bbd4bdbcb37f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46237 } }
I20260926 07:29:27.856844 22193 raft_consensus.cc:388] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:27.856922 22193 raft_consensus.cc:482] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:27.856923 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.849218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37651 (local address 127.0.98.62:46237)
0926 07:29:27.849645 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:27.849660 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:27.849695 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:29:27.850366 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:27.850384 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:27.850774 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:27.851127 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:27.851536 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.852548 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.853447 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.853925 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.854099 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.855092 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:27.855100 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:27.855117 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:27.855486 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:27.855547 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:27.855569 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:27.855711 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:27.855893 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 07:29:27.856026 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":123,"threads_started":1}
I20260926 07:29:27.857055 22193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:27.857839 22193 raft_consensus.cc:504] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab103b383a54be7878bbd4bdbcb37f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46237 } }
I20260926 07:29:27.858163 22193 leader_election.cc:302] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eab103b383a54be7878bbd4bdbcb37f5; no voters:
I20260926 07:29:27.858469 22193 leader_election.cc:288] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:27.858546 22198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:27.858742 22198 raft_consensus.cc:686] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [term 1 LEADER]: Becoming Leader. State: Replica: eab103b383a54be7878bbd4bdbcb37f5, State: Running, Role: LEADER
I20260926 07:29:27.859097 22198 consensus_queue.cc:237] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab103b383a54be7878bbd4bdbcb37f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46237 } }
I20260926 07:29:27.859719 22193 sys_catalog.cc:564] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:27.859853 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:27.860702 22200 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eab103b383a54be7878bbd4bdbcb37f5. Latest consensus state: current_term: 1 leader_uuid: "eab103b383a54be7878bbd4bdbcb37f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab103b383a54be7878bbd4bdbcb37f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46237 } } }
I20260926 07:29:27.860843 22200 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:27.861263 22207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:27.861213 22199 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eab103b383a54be7878bbd4bdbcb37f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab103b383a54be7878bbd4bdbcb37f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46237 } } }
I20260926 07:29:27.861398 22199 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:27.864466 22207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:27.866930 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:27.862247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55093 (local address 127.0.98.62:46237)
0926 07:29:27.862336 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:27.862340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:27.862578 (+ 238us) server_negotiation.cc:407] Connection header received
0926 07:29:27.862778 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:27.862781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:27.862820 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:27.862865 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:27.863279 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.864005 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.865097 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:27.865431 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:27.865556 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:27.866442 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:27.866444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:27.866446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:27.866457 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:27.866479 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:27.866488 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:27.866585 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:27.866764 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 07:29:27.866797 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 07:29:27.869916 22207 catalog_manager.cc:1347] Generated new cluster ID: b0a4d6ca13ab49a78fd74df73ce38e2c
I20260926 07:29:27.869974 22207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:27.875167 22207 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:27.876349 22207 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:27.885442 22207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eab103b383a54be7878bbd4bdbcb37f5: Generated new TSK 0
I20260926 07:29:27.886191 22207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:27.894512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 22125
W20260926 07:29:27.899816 22110 connection.cc:535] client connection to 127.0.98.62:46237 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46237): Connection reset by peer (error 104)
2026-09-26T07:29:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:27 dist-test-slave-g04g krb5kdc[22224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:27 dist-test-slave-g04g krb5kdc[22224](info): set up 2 sockets
Sep 26 07:29:27 dist-test-slave-g04g krb5kdc[22224](info): commencing operation
krb5kdc: starting...
W20260926 07:29:28.935333 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:29 dist-test-slave-g04g krb5kdc[22224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:29:29.058049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34437
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:29.167805 22237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:29.168056 22237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:29.168125 22237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:29.172135 22237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:29.172188 22237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:29.172207 22237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:29.172224 22237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:29.172276 22237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:29.177608 22237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:29.178620 22237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:29.179656 22237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:29.187829 22245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:29.187842 22243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:29.187947 22237 server_base.cc:1034] running on GCE node
W20260926 07:29:29.187810 22242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:29.188674 22237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:29.189730 22237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:29.190917 22237 hybrid_clock.cc:648] HybridClock initialized: now 1790407769190901 us; error 27 us; skew 500 ppm
Sep 26 07:29:29 dist-test-slave-g04g krb5kdc[22224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:29.197269 22237 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:29:29.198136 22237 webserver.cc:460] Webserver started at http://127.0.98.62:40401/ using document root <none> and password file <none>
I20260926 07:29:29.198704 22237 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:29.198794 22237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:29.199026 22237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:29.201104 22237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "2313b847b794494ba8d309bf664d3c44"
format_stamp: "Formatted at 2026-09-26 07:29:29 on dist-test-slave-g04g"
server_key: "15d92a9dfba159885c85d4524ecc55d5"
server_key_iv: "d5a48903524054aa2114b25cd85008bd"
server_key_version: "encryptionkey@0"
I20260926 07:29:29.201691 22237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "2313b847b794494ba8d309bf664d3c44"
format_stamp: "Formatted at 2026-09-26 07:29:29 on dist-test-slave-g04g"
server_key: "15d92a9dfba159885c85d4524ecc55d5"
server_key_iv: "d5a48903524054aa2114b25cd85008bd"
server_key_version: "encryptionkey@0"
I20260926 07:29:29.205411 22237 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:29:29.207799 22252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:29.208982 22237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:29.209124 22237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "2313b847b794494ba8d309bf664d3c44"
format_stamp: "Formatted at 2026-09-26 07:29:29 on dist-test-slave-g04g"
server_key: "15d92a9dfba159885c85d4524ecc55d5"
server_key_iv: "d5a48903524054aa2114b25cd85008bd"
server_key_version: "encryptionkey@0"
I20260926 07:29:29.209235 22237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:29.213656 22237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:29.216432 22237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:29.216637 22237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:29.223528 22237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34437
I20260926 07:29:29.223555 22304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34437 every 8 connection(s)
I20260926 07:29:29.224460 22237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:29.227234 22305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:29.232547 22305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44: Bootstrap starting.
I20260926 07:29:29.234007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 22237
I20260926 07:29:29.234161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:29.234428 392 external_mini_cluster.cc:1428] Setting key 3ff300b7d18b73a276affe7864e67fff
I20260926 07:29:29.235405 22305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:29.236235 22305 log.cc:824] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:29.238265 22305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44: No bootstrap required, opened a new log
I20260926 07:29:29.239596 22308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:29.235552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52009 (local address 127.0.98.62:34437)
0926 07:29:29.235985 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:29.236000 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:29.236034 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:29:29.236856 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:29.236882 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:29.237212 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:29.237554 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:29.238326 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:29.238332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:29.238350 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:29.238364 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:29.238428 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:29.238448 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:29.238562 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:29.238668 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 07:29:29.238828 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":121,"threads_started":1}
I20260926 07:29:29.242192 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:29.242014 22305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2313b847b794494ba8d309bf664d3c44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34437 } }
I20260926 07:29:29.242297 22305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:29.242367 22305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2313b847b794494ba8d309bf664d3c44, State: Initialized, Role: FOLLOWER
I20260926 07:29:29.242802 22305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2313b847b794494ba8d309bf664d3c44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34437 } }
I20260926 07:29:29.242918 22305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:29.243000 22305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:29.243120 22305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:29.244124 22305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2313b847b794494ba8d309bf664d3c44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34437 } }
I20260926 07:29:29.244506 22305 leader_election.cc:302] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2313b847b794494ba8d309bf664d3c44; no voters:
I20260926 07:29:29.244868 22305 leader_election.cc:288] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:29.244947 22315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:29.245167 22315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [term 1 LEADER]: Becoming Leader. State: Replica: 2313b847b794494ba8d309bf664d3c44, State: Running, Role: LEADER
I20260926 07:29:29.245496 22315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2313b847b794494ba8d309bf664d3c44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34437 } }
I20260926 07:29:29.246160 22308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:29.244345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34867 (local address 127.0.98.62:34437)
0926 07:29:29.244477 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:29.244482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:29.244829 (+ 347us) server_negotiation.cc:407] Connection header received
0926 07:29:29.244867 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:29.244869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:29.244904 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:29.244973 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:29.245783 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:29.245786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:29.245788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:29.245800 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:29.245826 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:29.245839 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:29.245925 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:29.246020 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 07:29:29.246054 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:29:29.246212 22305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:29.247268 22317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2313b847b794494ba8d309bf664d3c44. Latest consensus state: current_term: 1 leader_uuid: "2313b847b794494ba8d309bf664d3c44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2313b847b794494ba8d309bf664d3c44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34437 } } }
I20260926 07:29:29.247397 22317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:29.247685 22316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2313b847b794494ba8d309bf664d3c44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2313b847b794494ba8d309bf664d3c44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34437 } } }
I20260926 07:29:29.247745 22324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:29.247786 22316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:29.250597 22324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:29.255579 22324 catalog_manager.cc:1347] Generated new cluster ID: 672c966076834d5f9e72cf4d884d55d5
I20260926 07:29:29.255646 22324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:29.266830 22324 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:29.267170 22324 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:29.274315 22324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2313b847b794494ba8d309bf664d3c44: Generated new TSK 0
I20260926 07:29:29.274947 22324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:29.278684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 22237
2026-09-26T07:29:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:29 dist-test-slave-g04g krb5kdc[22336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:29 dist-test-slave-g04g krb5kdc[22336](info): set up 2 sockets
Sep 26 07:29:29 dist-test-slave-g04g krb5kdc[22336](info): commencing operation
krb5kdc: starting...
W20260926 07:29:30.315395 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:30 dist-test-slave-g04g krb5kdc[22336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:29:30.437939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42675
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:30.550443 22349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:30.550762 22349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:30.550848 22349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:30.554893 22349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:30.554991 22349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:30.555027 22349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:30.555080 22349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:30.555137 22349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:30.560518 22349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:30.561605 22349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:30.562681 22349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:30.571921 22355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:30.571939 22349 server_base.cc:1034] running on GCE node
W20260926 07:29:30.571985 22354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:30.572002 22357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:30.572790 22349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:30.573908 22349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:30.575157 22349 hybrid_clock.cc:648] HybridClock initialized: now 1790407770575141 us; error 31 us; skew 500 ppm
Sep 26 07:29:30 dist-test-slave-g04g krb5kdc[22336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:30.581066 22349 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:29:30.582023 22349 webserver.cc:460] Webserver started at http://127.0.98.62:33123/ using document root <none> and password file <none>
I20260926 07:29:30.582636 22349 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:30.582731 22349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:30.582967 22349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:30.585060 22349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "da35285710b24936a076b79d618ebb46"
format_stamp: "Formatted at 2026-09-26 07:29:30 on dist-test-slave-g04g"
server_key: "aca03b153204a9a68487ec7458a3c2e6"
server_key_iv: "81d80829be0c1e7b8c1595e047a10a23"
server_key_version: "encryptionkey@0"
I20260926 07:29:30.585646 22349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "da35285710b24936a076b79d618ebb46"
format_stamp: "Formatted at 2026-09-26 07:29:30 on dist-test-slave-g04g"
server_key: "aca03b153204a9a68487ec7458a3c2e6"
server_key_iv: "81d80829be0c1e7b8c1595e047a10a23"
server_key_version: "encryptionkey@0"
I20260926 07:29:30.589409 22349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:29:30.591969 22364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:30.593221 22349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:30.593370 22349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "da35285710b24936a076b79d618ebb46"
format_stamp: "Formatted at 2026-09-26 07:29:30 on dist-test-slave-g04g"
server_key: "aca03b153204a9a68487ec7458a3c2e6"
server_key_iv: "81d80829be0c1e7b8c1595e047a10a23"
server_key_version: "encryptionkey@0"
I20260926 07:29:30.593487 22349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:30.608830 22349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:30.611972 22349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:30.612193 22349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:30.619419 22349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42675
I20260926 07:29:30.619485 22416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42675 every 8 connection(s)
I20260926 07:29:30.620798 22349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:30.624289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 22349
I20260926 07:29:30.624431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:30.624580 22417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:30.624768 392 external_mini_cluster.cc:1428] Setting key 868a113f182e838caeadc65e7289e8cc
I20260926 07:29:30.631649 22417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46: Bootstrap starting.
I20260926 07:29:30.634220 22417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:30.634963 22417 log.cc:824] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46: Log is configured to *not* fsync() on all Append() calls
Sep 26 07:29:30 dist-test-slave-g04g krb5kdc[22336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:30.637292 22417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46: No bootstrap required, opened a new log
I20260926 07:29:30.641330 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:30.626760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46359 (local address 127.0.98.62:42675)
0926 07:29:30.627990 (+ 1230us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:30.628011 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:30.628054 (+ 43us) server_negotiation.cc:407] Connection header received
0926 07:29:30.628870 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:30.628896 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:30.629323 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:30.629689 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:30.630363 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:30.631824 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:30.632718 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:30.633384 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:30.633551 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:30.636409 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:30.636427 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:30.636446 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:30.636476 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:30.638505 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:30.639017 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:30.639022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:30.639026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:30.639206 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:30.639518 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:30.639520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:30.639522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:30.639912 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:30.640047 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:30.640364 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 07:29:30.640600 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":158,"threads_started":1}
I20260926 07:29:30.641865 22417 raft_consensus.cc:348] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da35285710b24936a076b79d618ebb46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42675 } }
I20260926 07:29:30.642120 22417 raft_consensus.cc:374] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:30.642205 22417 raft_consensus.cc:729] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da35285710b24936a076b79d618ebb46, State: Initialized, Role: FOLLOWER
I20260926 07:29:30.642692 22417 consensus_queue.cc:260] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da35285710b24936a076b79d618ebb46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42675 } }
I20260926 07:29:30.642848 22417 raft_consensus.cc:388] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:30.642923 22417 raft_consensus.cc:482] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:30.643016 22417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:30.644141 22417 raft_consensus.cc:504] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da35285710b24936a076b79d618ebb46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42675 } }
I20260926 07:29:30.644232 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:30.644491 22417 leader_election.cc:302] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da35285710b24936a076b79d618ebb46; no voters:
I20260926 07:29:30.644785 22417 leader_election.cc:288] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:30.644906 22422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:30.645147 22422 raft_consensus.cc:686] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [term 1 LEADER]: Becoming Leader. State: Replica: da35285710b24936a076b79d618ebb46, State: Running, Role: LEADER
I20260926 07:29:30.645534 22422 consensus_queue.cc:237] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da35285710b24936a076b79d618ebb46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42675 } }
I20260926 07:29:30.646071 22417 sys_catalog.cc:564] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:30.646848 22424 sys_catalog.cc:455] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da35285710b24936a076b79d618ebb46. Latest consensus state: current_term: 1 leader_uuid: "da35285710b24936a076b79d618ebb46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da35285710b24936a076b79d618ebb46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42675 } } }
I20260926 07:29:30.647056 22424 sys_catalog.cc:458] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:30.647287 22423 sys_catalog.cc:455] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da35285710b24936a076b79d618ebb46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da35285710b24936a076b79d618ebb46" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42675 } } }
I20260926 07:29:30.647464 22436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:30.647385 22423 sys_catalog.cc:458] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:30.650732 22436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:30.656873 22436 catalog_manager.cc:1347] Generated new cluster ID: 6ce8c2587bd5439a8133aaad20abdd8e
I20260926 07:29:30.656991 22436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:30.657382 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:30.646467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34065 (local address 127.0.98.62:42675)
0926 07:29:30.646584 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:30.646591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:30.648445 (+ 1854us) server_negotiation.cc:407] Connection header received
0926 07:29:30.648572 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:30.648578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:30.648636 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:30.648740 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:30.649567 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:30.650317 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:30.651555 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:30.651886 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:30.651992 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:30.653657 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:30.653661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:30.653664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:30.653690 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:30.655575 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:30.655993 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:30.655997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:30.655999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:30.656083 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:30.656364 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:30.656369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:30.656370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:30.656566 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:30.656702 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:30.657093 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 07:29:30.657221 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:29:30.665657 22436 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:30.666680 22436 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:30.673337 22436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da35285710b24936a076b79d618ebb46: Generated new TSK 0
I20260926 07:29:30.674113 22436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:30.683956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 22349
2026-09-26T07:29:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:30 dist-test-slave-g04g krb5kdc[22448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:30 dist-test-slave-g04g krb5kdc[22448](info): set up 2 sockets
Sep 26 07:29:30 dist-test-slave-g04g krb5kdc[22448](info): commencing operation
krb5kdc: starting...
W20260926 07:29:31.731340 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:31 dist-test-slave-g04g krb5kdc[22448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:29:31.854192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34201
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:31.963171 22461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:31.963469 22461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:31.963563 22461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:31.967792 22461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:31.967867 22461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:31.967926 22461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:31.967972 22461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:31.968010 22461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:31.973206 22461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:31.974229 22461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:31.975507 22461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:31.983879 22469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:31.983940 22467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:31.984117 22461 server_base.cc:1034] running on GCE node
W20260926 07:29:31.983947 22466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:31.984831 22461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:31.985873 22461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:31.987097 22461 hybrid_clock.cc:648] HybridClock initialized: now 1790407771987084 us; error 28 us; skew 500 ppm
Sep 26 07:29:31 dist-test-slave-g04g krb5kdc[22448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:31.989823 22461 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:29:31.990648 22461 webserver.cc:460] Webserver started at http://127.0.98.62:38403/ using document root <none> and password file <none>
I20260926 07:29:31.991264 22461 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:31.991353 22461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:31.991580 22461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:31.993535 22461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "41ba866ff2ed49babc74daf27aa14b66"
format_stamp: "Formatted at 2026-09-26 07:29:31 on dist-test-slave-g04g"
server_key: "7205e28a161d11d8f4c5a6e914721c40"
server_key_iv: "6d801c9353428ef35b43129f6f9960d1"
server_key_version: "encryptionkey@0"
I20260926 07:29:31.994092 22461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "41ba866ff2ed49babc74daf27aa14b66"
format_stamp: "Formatted at 2026-09-26 07:29:31 on dist-test-slave-g04g"
server_key: "7205e28a161d11d8f4c5a6e914721c40"
server_key_iv: "6d801c9353428ef35b43129f6f9960d1"
server_key_version: "encryptionkey@0"
I20260926 07:29:31.997635 22461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 07:29:31.999971 22476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:32.001120 22461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:32.001252 22461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "41ba866ff2ed49babc74daf27aa14b66"
format_stamp: "Formatted at 2026-09-26 07:29:31 on dist-test-slave-g04g"
server_key: "7205e28a161d11d8f4c5a6e914721c40"
server_key_iv: "6d801c9353428ef35b43129f6f9960d1"
server_key_version: "encryptionkey@0"
I20260926 07:29:32.001358 22461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:32.014642 22461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:32.017206 22461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:32.017392 22461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:32.023910 22461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34201
I20260926 07:29:32.023936 22528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34201 every 8 connection(s)
I20260926 07:29:32.024780 22461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:32.027383 22529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:32.030063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 22461
I20260926 07:29:32.030189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:32.030493 392 external_mini_cluster.cc:1428] Setting key 582fc8a03c373bf2deef8cc33e58366a
I20260926 07:29:32.033252 22529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66: Bootstrap starting.
I20260926 07:29:32.035390 22529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:32.036078 22529 log.cc:824] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:32.038049 22529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66: No bootstrap required, opened a new log
I20260926 07:29:32.038879 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:32.031873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45561 (local address 127.0.98.62:34201)
0926 07:29:32.032280 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:32.032293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:32.032326 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:29:32.032909 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:32.032925 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:32.033234 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:32.033598 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:32.033985 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:32.034932 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:32.035905 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:32.036436 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:32.036628 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:32.037344 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:32.037349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:32.037360 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:32.037726 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:32.037775 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:32.037797 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:32.037867 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:32.038033 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 07:29:32.038234 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
I20260926 07:29:32.041875 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:32.042119 22529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ba866ff2ed49babc74daf27aa14b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34201 } }
I20260926 07:29:32.042325 22529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:32.042405 22529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ba866ff2ed49babc74daf27aa14b66, State: Initialized, Role: FOLLOWER
I20260926 07:29:32.042831 22529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ba866ff2ed49babc74daf27aa14b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34201 } }
I20260926 07:29:32.042948 22529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:32.043054 22529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:32.043157 22529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:32.044034 22529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ba866ff2ed49babc74daf27aa14b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34201 } }
I20260926 07:29:32.044440 22529 leader_election.cc:302] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41ba866ff2ed49babc74daf27aa14b66; no voters:
I20260926 07:29:32.044806 22529 leader_election.cc:288] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:32.044902 22539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:32.045121 22539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [term 1 LEADER]: Becoming Leader. State: Replica: 41ba866ff2ed49babc74daf27aa14b66, State: Running, Role: LEADER
I20260926 07:29:32.045460 22539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ba866ff2ed49babc74daf27aa14b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34201 } }
I20260926 07:29:32.046051 22529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:32.047173 22540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41ba866ff2ed49babc74daf27aa14b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ba866ff2ed49babc74daf27aa14b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34201 } } }
I20260926 07:29:32.047304 22540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:32.047650 22541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41ba866ff2ed49babc74daf27aa14b66. Latest consensus state: current_term: 1 leader_uuid: "41ba866ff2ed49babc74daf27aa14b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41ba866ff2ed49babc74daf27aa14b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34201 } } }
I20260926 07:29:32.047775 22541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:32.048763 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:32.044026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38257 (local address 127.0.98.62:34201)
0926 07:29:32.044146 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:32.044150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:32.044334 (+ 184us) server_negotiation.cc:407] Connection header received
0926 07:29:32.044443 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:32.044445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:32.044480 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:32.044542 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:32.044849 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:32.045581 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:32.046643 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:32.046971 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:32.047105 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:32.048023 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:32.048029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:32.048032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:32.048045 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:32.048070 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:32.048081 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:32.048157 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:32.048594 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 07:29:32.048649 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 07:29:32.050925 22554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:29:32.050987 22554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:32.051080 22548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:32.051803 22548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:32.057367 22548 catalog_manager.cc:1347] Generated new cluster ID: dfb6e8e7e77b45dd834254ac09294a03
I20260926 07:29:32.057435 22548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:32.068393 22548 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:32.069238 22548 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:32.073597 22548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41ba866ff2ed49babc74daf27aa14b66: Generated new TSK 0
I20260926 07:29:32.074222 22548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:32.079386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 22461
2026-09-26T07:29:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:32 dist-test-slave-g04g krb5kdc[22560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:32 dist-test-slave-g04g krb5kdc[22560](info): set up 2 sockets
Sep 26 07:29:32 dist-test-slave-g04g krb5kdc[22560](info): commencing operation
krb5kdc: starting...
W20260926 07:29:33.111281 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:33 dist-test-slave-g04g krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:29:33.233685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33089
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:33.341041 22573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:33.341277 22573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:33.341347 22573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:33.345321 22573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:33.345379 22573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:33.345422 22573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:33.345458 22573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:33.345494 22573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:33.350630 22573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:33.351647 22573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:33.352619 22573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:33.361217 22579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:33.361279 22573 server_base.cc:1034] running on GCE node
W20260926 07:29:33.361217 22578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:33.361232 22581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:33.361976 22573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:33.363022 22573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:33.364266 22573 hybrid_clock.cc:648] HybridClock initialized: now 1790407773364247 us; error 30 us; skew 500 ppm
Sep 26 07:29:33 dist-test-slave-g04g krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:33.366876 22573 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:29:33.367739 22573 webserver.cc:460] Webserver started at http://127.0.98.62:45881/ using document root <none> and password file <none>
I20260926 07:29:33.368278 22573 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:33.368366 22573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:33.368597 22573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:33.370548 22573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "9f614bae5a904135a8bf9adaaccf8f61"
format_stamp: "Formatted at 2026-09-26 07:29:33 on dist-test-slave-g04g"
server_key: "2b392fd4fbcb981e05b44563e3a3c9cc"
server_key_iv: "0fed184b20943627c32500eddc218254"
server_key_version: "encryptionkey@0"
I20260926 07:29:33.371176 22573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f614bae5a904135a8bf9adaaccf8f61"
format_stamp: "Formatted at 2026-09-26 07:29:33 on dist-test-slave-g04g"
server_key: "2b392fd4fbcb981e05b44563e3a3c9cc"
server_key_iv: "0fed184b20943627c32500eddc218254"
server_key_version: "encryptionkey@0"
I20260926 07:29:33.374588 22573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 07:29:33.376893 22588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:33.378062 22573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:33.378197 22573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "9f614bae5a904135a8bf9adaaccf8f61"
format_stamp: "Formatted at 2026-09-26 07:29:33 on dist-test-slave-g04g"
server_key: "2b392fd4fbcb981e05b44563e3a3c9cc"
server_key_iv: "0fed184b20943627c32500eddc218254"
server_key_version: "encryptionkey@0"
I20260926 07:29:33.378302 22573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:33.382618 22573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:33.385389 22573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:33.385582 22573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:33.392184 22573 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33089
I20260926 07:29:33.392253 22640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33089 every 8 connection(s)
I20260926 07:29:33.393069 22573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:33.395707 22641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:33.399163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 22573
I20260926 07:29:33.399292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:33.399629 392 external_mini_cluster.cc:1428] Setting key 011305fed1e1b2342f9e6f49c989e3e6
I20260926 07:29:33.401414 22641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61: Bootstrap starting.
I20260926 07:29:33.403633 22641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:33.404404 22641 log.cc:824] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:33.404546 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:33.400948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53815 (local address 127.0.98.62:33089)
0926 07:29:33.401365 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:33.401378 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:33.401412 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:29:33.402048 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:33.402063 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:33.402339 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:33.402664 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:33.403381 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:33.403387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:33.403405 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:33.403414 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:33.403457 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:33.403474 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:33.403587 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:33.403704 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 07:29:33.403849 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":123,"threads_started":1}
I20260926 07:29:33.406328 22641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61: No bootstrap required, opened a new log
I20260926 07:29:33.407202 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:33.410241 22641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f614bae5a904135a8bf9adaaccf8f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } }
I20260926 07:29:33.410506 22641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:29:33.410585 22641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f614bae5a904135a8bf9adaaccf8f61, State: Initialized, Role: FOLLOWER
I20260926 07:29:33.410609 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:33.409086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44685 (local address 127.0.98.62:33089)
0926 07:29:33.409194 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:33.409200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:33.409397 (+ 197us) server_negotiation.cc:407] Connection header received
0926 07:29:33.409486 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:33.409489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:33.409521 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:33.409601 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:33.410241 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:33.410244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:33.410246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:29:33.410260 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:33.410289 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:29:33.410302 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:33.410394 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:33.410428 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 07:29:33.410461 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 07:29:33.411124 22641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f614bae5a904135a8bf9adaaccf8f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } }
I20260926 07:29:33.411255 22641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:29:33.411334 22641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:29:33.411445 22641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:33.412298 22641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f614bae5a904135a8bf9adaaccf8f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } }
I20260926 07:29:33.412622 22641 leader_election.cc:302] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f614bae5a904135a8bf9adaaccf8f61; no voters:
I20260926 07:29:33.412899 22641 leader_election.cc:288] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:29:33.413004 22651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:33.413231 22651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [term 1 LEADER]: Becoming Leader. State: Replica: 9f614bae5a904135a8bf9adaaccf8f61, State: Running, Role: LEADER
I20260926 07:29:33.413547 22651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f614bae5a904135a8bf9adaaccf8f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } }
I20260926 07:29:33.414039 22641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:33.415221 22653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f614bae5a904135a8bf9adaaccf8f61. Latest consensus state: current_term: 1 leader_uuid: "9f614bae5a904135a8bf9adaaccf8f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f614bae5a904135a8bf9adaaccf8f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } }
I20260926 07:29:33.415364 22653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:33.415201 22652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f614bae5a904135a8bf9adaaccf8f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f614bae5a904135a8bf9adaaccf8f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33089 } } }
I20260926 07:29:33.415669 22652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61 [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:33.415867 22661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:33.418645 22661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:33.423691 22661 catalog_manager.cc:1347] Generated new cluster ID: 8c97863dfb5e4b4d891105875e8a7e62
I20260926 07:29:33.423754 22661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:33.450794 22661 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:33.451205 22661 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:33.458240 22661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f614bae5a904135a8bf9adaaccf8f61: Generated new TSK 0
I20260926 07:29:33.459015 22661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:29:33.470779 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 22573
2026-09-26T07:29:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1391 ms)
[----------] 18 tests from AuthTokenIssuingTest (25429 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:33 dist-test-slave-g04g krb5kdc[22672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:33 dist-test-slave-g04g krb5kdc[22672](info): set up 2 sockets
Sep 26 07:29:33 dist-test-slave-g04g krb5kdc[22672](info): commencing operation
krb5kdc: starting...
W20260926 07:29:34.503360 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:34 dist-test-slave-g04g krb5kdc[22672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:29:34.626487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42121
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42121,127.0.98.61:41639,127.0.98.60:35117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:34.737128 22685 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:29:34.737425 22685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:34.737448 22685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:34.737547 22685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:34.741578 22685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:34.741638 22685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:34.741657 22685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:34.741721 22685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:34.741755 22685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:34.747018 22685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42121,127.0.98.61:41639,127.0.98.60:35117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:34.748056 22685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:34.749078 22685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:34.757337 22690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:34.757433 22685 server_base.cc:1034] running on GCE node
W20260926 07:29:34.757373 22691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:34.757380 22693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:34.758127 22685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:34.759284 22685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:34.760443 22685 hybrid_clock.cc:648] HybridClock initialized: now 1790407774760416 us; error 42 us; skew 500 ppm
Sep 26 07:29:34 dist-test-slave-g04g krb5kdc[22672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:34.766417 22685 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:29:34.767392 22685 webserver.cc:460] Webserver started at http://127.0.98.62:37463/ using document root <none> and password file <none>
I20260926 07:29:34.767915 22685 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:34.767971 22685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:34.768141 22685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:34.770126 22685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "6944f46bd31e4564b63244b602da746e"
format_stamp: "Formatted at 2026-09-26 07:29:34 on dist-test-slave-g04g"
server_key: "e188c88638dc9d23ffdb6d281dad38ec"
server_key_iv: "0bc65e45a256ca4237e99b2e395d5880"
server_key_version: "encryptionkey@0"
I20260926 07:29:34.770635 22685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "6944f46bd31e4564b63244b602da746e"
format_stamp: "Formatted at 2026-09-26 07:29:34 on dist-test-slave-g04g"
server_key: "e188c88638dc9d23ffdb6d281dad38ec"
server_key_iv: "0bc65e45a256ca4237e99b2e395d5880"
server_key_version: "encryptionkey@0"
I20260926 07:29:34.774260 22685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 07:29:34.776665 22700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:34.777865 22685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:29:34.778007 22685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "6944f46bd31e4564b63244b602da746e"
format_stamp: "Formatted at 2026-09-26 07:29:34 on dist-test-slave-g04g"
server_key: "e188c88638dc9d23ffdb6d281dad38ec"
server_key_iv: "0bc65e45a256ca4237e99b2e395d5880"
server_key_version: "encryptionkey@0"
I20260926 07:29:34.778133 22685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:34.792039 22685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:34.795360 22685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:34.795570 22685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:34.802469 22685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42121
I20260926 07:29:34.802767 22752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42121 every 8 connection(s)
I20260926 07:29:34.803525 22685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:34.806211 22753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:34.809722 22753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:29:34.813027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 22685
I20260926 07:29:34.813158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:34.813441 392 external_mini_cluster.cc:1428] Setting key cba2e2ac12f6b709d5f14702378712c6
Sep 26 07:29:34 dist-test-slave-g04g krb5kdc[22672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260926 07:29:34.824712 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:34.811803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42121 (local address 127.0.0.1:56590)
0926 07:29:34.812273 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:34.812311 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:34.813068 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:34.813852 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:34.813857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:34.814281 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:34.814519 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:34.814532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:34.815751 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:34.815754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:34.816455 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:34.816464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:34.817130 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:34.817134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:34.817242 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:34.817868 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:34.817884 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:34.819696 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:34.821666 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:34.821674 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:34.821681 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:34.821899 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:34.822320 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:34.822323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:34.822325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:34.822433 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:34.823138 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:34.823144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:34.823282 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 07:29:34.823515 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 07:29:34.823990 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":123,"spinlock_wait_cycles":62208,"thread_start_us":154,"threads_started":1}
I20260926 07:29:34.824717 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:34.811803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.0.98.62:42121)
0926 07:29:34.812208 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:34.812225 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:34.812412 (+ 187us) server_negotiation.cc:407] Connection header received
0926 07:29:34.813425 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:34.813430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:34.813670 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:34.813852 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:34.814674 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:34.815629 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:34.816608 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:34.816973 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:34.817156 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:34.819847 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:34.819852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:34.819862 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:34.819882 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:34.821564 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:34.822109 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:34.822113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:34.822114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:34.822215 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:34.822534 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:34.822537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:34.822539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:34.822959 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:34.823167 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:34.823630 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 07:29:34.823901 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":278,"thread_start_us":114,"threads_started":1}
I20260926 07:29:34.826972 22753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:29:34.827740 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:34.827503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41639 (local address 127.0.0.1:33506)
0926 07:29:34.827618 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:34.827683 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:29:34.827998 22701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41639: connect: Connection refused (error 111)
I20260926 07:29:34.828833 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:34.828641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41639 (local address 127.0.0.1:33518)
0926 07:29:34.828742 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:34.828783 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:29:34.829075 22753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41639: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41639: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 07:29:34.840162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41639
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42121,127.0.98.61:41639,127.0.98.60:35117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:29:34.878293 22753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } attempt: 1
I20260926 07:29:34.879238 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:34.878850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41639 (local address 127.0.0.1:33532)
0926 07:29:34.879058 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:34.879145 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 07:29:34.880016 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:34.879766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41639 (local address 127.0.0.1:33540)
0926 07:29:34.879883 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:34.879938 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:29:34.880180 22753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41639: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41639: connect: Connection refused (error 111)
I20260926 07:29:34.948360 22753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } attempt: 2
I20260926 07:29:34.949282 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:34.948912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41639 (local address 127.0.0.1:33548)
0926 07:29:34.949062 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:34.949169 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 07:29:34.949987 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:34.949761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41639 (local address 127.0.0.1:33554)
0926 07:29:34.949875 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:34.949930 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:29:34.950120 22753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41639: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41639: connect: Connection refused (error 111)
W20260926 07:29:34.950795 22760 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:29:34.951121 22760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:34.951193 22760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:34.951265 22760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:34.955268 22760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:34.955338 22760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:34.955382 22760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:34.955418 22760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:34.955449 22760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:34.961047 22760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42121,127.0.98.61:41639,127.0.98.60:35117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:34.962102 22760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:34.963119 22760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:34.971352 22768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:34.971422 22766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:34.971386 22765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:34.971503 22760 server_base.cc:1034] running on GCE node
I20260926 07:29:34.972327 22760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:34.973511 22760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:34.974696 22760 hybrid_clock.cc:648] HybridClock initialized: now 1790407774974669 us; error 50 us; skew 500 ppm
Sep 26 07:29:34 dist-test-slave-g04g krb5kdc[22672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:34.977475 22760 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 07:29:34.978350 22760 webserver.cc:460] Webserver started at http://127.0.98.61:44063/ using document root <none> and password file <none>
I20260926 07:29:34.978888 22760 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:34.978976 22760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:34.979264 22760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:34.981323 22760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/data/instance:
uuid: "d98e34c12c1d4ffd90e0439c60b31ad4"
format_stamp: "Formatted at 2026-09-26 07:29:34 on dist-test-slave-g04g"
server_key: "12ef0766d7ee57b314a7235daad1028f"
server_key_iv: "bbd42822ca526e65a8c4d318159d92e4"
server_key_version: "encryptionkey@0"
I20260926 07:29:34.981858 22760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/wal/instance:
uuid: "d98e34c12c1d4ffd90e0439c60b31ad4"
format_stamp: "Formatted at 2026-09-26 07:29:34 on dist-test-slave-g04g"
server_key: "12ef0766d7ee57b314a7235daad1028f"
server_key_iv: "bbd42822ca526e65a8c4d318159d92e4"
server_key_version: "encryptionkey@0"
I20260926 07:29:34.985558 22760 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260926 07:29:34.987895 22775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:34.989066 22760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:34.989212 22760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/wal
uuid: "d98e34c12c1d4ffd90e0439c60b31ad4"
format_stamp: "Formatted at 2026-09-26 07:29:34 on dist-test-slave-g04g"
server_key: "12ef0766d7ee57b314a7235daad1028f"
server_key_iv: "bbd42822ca526e65a8c4d318159d92e4"
server_key_version: "encryptionkey@0"
I20260926 07:29:34.989326 22760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:35.009366 22760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:35.012634 22760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:35.012840 22760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:35.019686 22760 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41639
I20260926 07:29:35.019963 22827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41639 every 8 connection(s)
I20260926 07:29:35.020638 22760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/data/info.pb
I20260926 07:29:35.023340 22828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:35.026013 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 22760
I20260926 07:29:35.026194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-1/wal/instance
I20260926 07:29:35.026458 392 external_mini_cluster.cc:1428] Setting key 38c52d4cfdc47d993e8d097780fb28a5
I20260926 07:29:35.027155 22828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 07:29:35 dist-test-slave-g04g krb5kdc[22672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:35.041286 22753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } attempt: 3
I20260926 07:29:35.041592 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.029009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.0.98.62:42121)
0926 07:29:35.029171 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.029176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.030065 (+ 889us) server_negotiation.cc:407] Connection header received
0926 07:29:35.031082 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.031090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.031159 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.031270 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.032399 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.033270 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.034382 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.034639 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.034757 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.037921 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.037924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.037927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.037951 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.039484 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.040020 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.040024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.040026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.040114 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.040497 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.040501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.040503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.040687 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.040794 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.041299 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.041422 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:29:35.042548 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.029411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42121 (local address 127.0.0.1:56606)
0926 07:29:35.029882 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.029932 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:35.030843 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:35.031427 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:35.031434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:35.031782 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:35.032242 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.032255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.033400 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.033403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.034252 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.034261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.034793 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.034797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.034963 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.035571 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:35.035586 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:35.037748 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:35.039643 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.039648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.039663 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.039913 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.040207 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.040212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.040214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.040404 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.040793 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:35.040801 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:35.041052 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 07:29:35.041234 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 07:29:35.041549 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":148,"threads_started":1}
I20260926 07:29:35.044145 22828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:29:35 dist-test-slave-g04g krb5kdc[22672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 07:29:35 dist-test-slave-g04g krb5kdc[22672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 07:29:35.053485 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.041744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41639 (local address 127.0.0.1:33566)
0926 07:29:35.041867 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.041881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:35.041954 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:35.042894 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:35.042897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:35.043096 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:35.043654 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.043663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.045179 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.045183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.045885 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.045894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.046562 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.046565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.046665 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.047137 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:35.047152 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:35.048730 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:35.051792 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.051795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.051796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.051981 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.052352 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.052356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.052358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.052444 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.052940 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:35.052943 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:35.053059 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 07:29:35.053183 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 07:29:35.053309 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 07:29:35.053610 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.044649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41639 (local address 127.0.0.1:33570)
0926 07:29:35.044806 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.044821 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:35.044885 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:35.045203 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:35.045206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:35.045364 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:35.045464 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.045470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.046244 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.046250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.046981 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.046990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.047545 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.047549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.047670 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.048081 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:35.048091 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:35.049258 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:35.052108 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.052112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.052135 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.052307 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.052577 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.052582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.052584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.052686 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.053039 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:35.053044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:35.053135 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 07:29:35.053237 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 07:29:35.053394 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":27}
I20260926 07:29:35.053644 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.042102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.0.98.61:41639)
0926 07:29:35.042466 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.042477 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.042499 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:29:35.042541 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.042548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.042747 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.043757 (+ 1010us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.043892 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.045042 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.046013 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.046451 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.046559 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.048856 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.048865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.048879 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.048904 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.051693 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.052137 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.052146 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.052151 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.052257 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.052543 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.052547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.052552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.052831 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.053017 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.053268 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.053455 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":261,"thread_start_us":150,"threads_started":1}
I20260926 07:29:35.054028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35117
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42121,127.0.98.61:41639,127.0.98.60:35117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:29:35.055565 22753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:29:35.056052 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.044682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.0.98.61:41639)
0926 07:29:35.044961 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.044969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.044982 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:29:35.045017 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.045022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.045065 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.045177 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.045576 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.046141 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.047165 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.047439 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.047549 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.049431 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.049437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.049441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.049466 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.051693 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.052402 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.052409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.052411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.052490 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.052775 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.052779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.052781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.052941 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.055621 (+ 2680us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.055712 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.055835 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":217,"thread_start_us":90,"threads_started":1}
I20260926 07:29:35.056267 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.056021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42028)
0926 07:29:35.056153 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.056203 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:29:35.056883 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.056666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42032)
0926 07:29:35.056785 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.056828 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 07:29:35.056931 22828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } has no permanent_uuid. Determining permanent_uuid...
W20260926 07:29:35.057091 22753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35117: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
I20260926 07:29:35.057576 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.057328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42042)
0926 07:29:35.057458 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.057527 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 07:29:35.057821 22777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
I20260926 07:29:35.058655 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.058471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42044)
0926 07:29:35.058570 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.058611 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:29:35.058861 22828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35117: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
I20260926 07:29:35.088269 22753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } attempt: 1
I20260926 07:29:35.089083 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.088804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42046)
0926 07:29:35.088947 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.089006 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 07:29:35.089838 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.089633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42062)
0926 07:29:35.089735 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.089780 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 07:29:35.090003 22753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35117: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
I20260926 07:29:35.108172 22828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } attempt: 1
I20260926 07:29:35.109018 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.108677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42078)
0926 07:29:35.108838 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.108901 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 07:29:35.109833 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.109614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42090)
0926 07:29:35.109730 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.109780 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:29:35.110073 22828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35117: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
I20260926 07:29:35.165194 22753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } attempt: 2
I20260926 07:29:35.166074 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.165778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42092)
0926 07:29:35.165931 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.165996 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 07:29:35.166095 22835 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:29:35.166477 22835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:35.166538 22835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:35.166613 22835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 07:29:35.166960 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.166769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42100)
0926 07:29:35.166865 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.166908 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 07:29:35.167117 22753 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35117: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
W20260926 07:29:35.170818 22835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:35.170871 22835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:35.170892 22835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:35.170943 22835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:35.170977 22835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:35.176290 22835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42121,127.0.98.61:41639,127.0.98.60:35117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:35.177325 22835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:35.178257 22828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } attempt: 2
I20260926 07:29:35.178400 22835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:29:35.179001 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.178747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42108)
0926 07:29:35.178871 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.178933 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 07:29:35.179759 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.179511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42116)
0926 07:29:35.179628 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.179684 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 07:29:35.179922 22828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35117: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35117: connect: Connection refused (error 111)
W20260926 07:29:35.186988 22844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:35.187062 22835 server_base.cc:1034] running on GCE node
W20260926 07:29:35.186995 22842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:35.186995 22841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:35.187794 22835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:35.188930 22835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:35.190086 22835 hybrid_clock.cc:648] HybridClock initialized: now 1790407775190068 us; error 30 us; skew 500 ppm
Sep 26 07:29:35 dist-test-slave-g04g krb5kdc[22672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:35.193313 22835 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 07:29:35.194180 22835 webserver.cc:460] Webserver started at http://127.0.98.60:44815/ using document root <none> and password file <none>
I20260926 07:29:35.194736 22835 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:35.194794 22835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:35.195012 22835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:35.197096 22835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/data/instance:
uuid: "aa773fdbbce54de18cc405123c193305"
format_stamp: "Formatted at 2026-09-26 07:29:35 on dist-test-slave-g04g"
server_key: "0dd499d8fbb000f3d2708da0e6d10306"
server_key_iv: "633dd0d6df605cc7abb511859b122c4a"
server_key_version: "encryptionkey@0"
I20260926 07:29:35.197664 22835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/wal/instance:
uuid: "aa773fdbbce54de18cc405123c193305"
format_stamp: "Formatted at 2026-09-26 07:29:35 on dist-test-slave-g04g"
server_key: "0dd499d8fbb000f3d2708da0e6d10306"
server_key_iv: "633dd0d6df605cc7abb511859b122c4a"
server_key_version: "encryptionkey@0"
I20260926 07:29:35.201442 22835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:29:35.203934 22851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:35.205060 22835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:35.205188 22835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/wal
uuid: "aa773fdbbce54de18cc405123c193305"
format_stamp: "Formatted at 2026-09-26 07:29:35 on dist-test-slave-g04g"
server_key: "0dd499d8fbb000f3d2708da0e6d10306"
server_key_iv: "633dd0d6df605cc7abb511859b122c4a"
server_key_version: "encryptionkey@0"
I20260926 07:29:35.205296 22835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:35.222321 22835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:35.225466 22835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:35.225667 22835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:35.232890 22835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35117
I20260926 07:29:35.233155 22903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35117 every 8 connection(s)
I20260926 07:29:35.233956 22835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/data/info.pb
I20260926 07:29:35.236639 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:35.240198 22904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:29:35.240715 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 22835
I20260926 07:29:35.240833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790407489657243-392-0/minicluster-data/master-2/wal/instance
I20260926 07:29:35.241120 392 external_mini_cluster.cc:1428] Setting key 27feb3f2d19a2ad9f85aa78accfb292c
Sep 26 07:29:35 dist-test-slave-g04g krb5kdc[22672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 07:29:35 dist-test-slave-g04g krb5kdc[22672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 07:29:35.253326 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.242246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.0.98.62:42121)
0926 07:29:35.242386 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.242390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.243097 (+ 707us) server_negotiation.cc:407] Connection header received
0926 07:29:35.243904 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.243907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.243953 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.244024 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.244918 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.245417 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.246520 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.246848 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.246936 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.249651 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.249655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.249657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.249681 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.251368 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.251874 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.251878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.251879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.251961 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.252301 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.252306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.252308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.252483 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.252591 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.253037 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.253150 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 07:29:35.254128 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.242361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42121 (local address 127.0.0.1:56620)
0926 07:29:35.242916 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.242960 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:35.243682 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:35.244105 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:35.244110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:35.244522 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:35.244757 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.244769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.245526 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.245529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.246407 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.246416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.246989 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.246992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.247248 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.247864 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:35.247879 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:35.249489 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:35.251478 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.251489 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.251498 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.251765 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.252060 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.252063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.252065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.252205 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.252587 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:35.252594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:35.252813 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 07:29:35.252981 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 07:29:35.253263 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"spinlock_wait_cycles":69376,"thread_start_us":161,"threads_started":1}
I20260926 07:29:35.254136 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.242361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.0.98.60:35117)
0926 07:29:35.242813 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.242830 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.242869 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:29:35.243682 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.243718 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.243948 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.244134 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.244878 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.245899 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.246842 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.247353 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.247475 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.250044 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.250050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.250061 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.250080 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.251656 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.252071 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.252078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.252080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.252200 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.252475 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.252479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.252481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.252799 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.252984 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.253231 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.253365 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":127,"threads_started":1}
I20260926 07:29:35.255326 22904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:29:35 dist-test-slave-g04g krb5kdc[22672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 07:29:35 dist-test-slave-g04g krb5kdc[22672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:35.263880 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.255749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41639 (local address 127.0.0.1:33574)
0926 07:29:35.255856 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.255874 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:35.255937 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:35.256173 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:35.256175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:35.256334 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:35.256438 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.256444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.257399 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.257406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.257907 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.257915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.258397 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.258401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.258492 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.258960 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:35.258973 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:35.260488 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:35.262389 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.262393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.262395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.262582 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.262871 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.262874 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.262876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.262957 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.263371 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:35.263374 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:35.263467 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 07:29:35.263572 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:29:35.263717 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 07:29:35.263885 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.255860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.0.98.61:41639)
0926 07:29:35.255990 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.255994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.256005 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:29:35.256040 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.256042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.256079 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.256146 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.256549 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.257294 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.258027 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.258288 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.258400 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.260622 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.260626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.260628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.260653 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.262265 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.262681 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.262684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.262686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.262775 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.263070 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.263074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.263076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.263267 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.263370 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.263622 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.263749 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:29:35.264952 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.256749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.0.98.62:42121)
0926 07:29:35.256855 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.256858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.256871 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:29:35.256953 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.256956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.256992 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.257066 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.257614 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.258073 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.258913 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.259276 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.259401 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.261646 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.261649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.261651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.261671 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.263234 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.263665 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.263669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.263671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.263765 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.264108 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.264113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.264116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.264331 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.264572 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.264675 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.264807 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:29:35.264942 22904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:29:35.266299 22753 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } attempt: 3
Sep 26 07:29:35 dist-test-slave-g04g krb5kdc[22672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 07:29:35.271070 22828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } attempt: 3
Sep 26 07:29:35 dist-test-slave-g04g krb5kdc[22672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 07:29:35 dist-test-slave-g04g krb5kdc[22672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 07:29:35.273880 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.265388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42132)
0926 07:29:35.265471 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.265485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:35.265552 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:35.265764 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:35.265767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:35.265923 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:35.265987 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.265992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.267131 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.267135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.267921 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.267930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.268532 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.268538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.268638 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.269093 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:35.269110 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:35.270280 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:35.272206 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.272211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.272213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.272439 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.272719 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.272721 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.272723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.272825 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.273213 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:35.273219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:35.273319 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 07:29:35.273427 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 07:29:35.273697 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 07:29:35.274912 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.265455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.0.98.60:35117)
0926 07:29:35.265573 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.265579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.265590 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:29:35.265621 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.265623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.265664 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.265722 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.266295 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.267013 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.268020 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.268413 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.268514 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.270404 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.270407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.270409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.270430 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.272122 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.272540 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.272546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.272548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.272625 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.272904 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.272909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.272914 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.273114 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.273202 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.273648 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.274737 (+ 1089us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":946,"server-negotiator.queue_time_us":55}
I20260926 07:29:35.276662 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.266641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42148)
0926 07:29:35.266748 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.266761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:35.266829 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:35.267391 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:35.267394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:35.267559 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:35.267645 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.267650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.269117 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.269120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.269935 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.269947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.270647 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.270653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.270763 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.271332 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:35.271347 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:35.272677 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:35.275114 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.275117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.275120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.275356 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.275650 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.275653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.275655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.275745 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.276191 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:35.276196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:35.276291 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 07:29:35.276366 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:29:35.276485 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 07:29:35.277277 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.266115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.0.98.61:41639)
0926 07:29:35.266211 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.266215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.266228 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:29:35.266296 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.266298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.266339 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.266412 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.267231 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.267807 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.268633 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.268988 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.269091 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.272868 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.272873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.272877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.272907 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.274902 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.275915 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.275918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.275920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.276003 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.276312 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.276316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.276317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.276510 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.276671 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.277020 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.277120 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 07:29:35.278071 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.266826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.0.98.60:35117)
0926 07:29:35.267180 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.267188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.267201 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:29:35.267231 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.267233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.267269 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.267350 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.267759 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.268971 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.270078 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.270497 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.270977 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.272829 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.272835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.272839 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.272860 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.274974 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.275477 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.275480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.275481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.275557 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.275847 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.275849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.275852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.276081 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.277151 (+ 1070us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.277226 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.277625 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
Sep 26 07:29:35 dist-test-slave-g04g krb5kdc[22672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 07:29:35.278309 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305: Bootstrap starting.
I20260926 07:29:35.281261 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:35.281262 22753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e: Bootstrap starting.
I20260926 07:29:35.281984 22904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:35.282377 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.271450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35117 (local address 127.0.0.1:42162)
0926 07:29:35.271690 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:35.271703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:35.271773 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:35.273202 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:35.273206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:35.273400 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:35.273501 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.273511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.275092 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.275098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.275887 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:35.275898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.276627 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.276630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:35.276720 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.277176 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:35.277204 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:35.278565 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:35.280807 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.280810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.280812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.281041 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.281411 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:35.281414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:35.281416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:35.281497 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:35.281898 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:35.281901 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:35.281988 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 07:29:35.282067 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:29:35.282198 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260926 07:29:35.282886 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.271909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.0.98.60:35117)
0926 07:29:35.272210 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.272218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.272231 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:29:35.272263 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.272266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.272304 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.273622 (+ 1318us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.273737 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.274533 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.276013 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.276507 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.278689 (+ 2182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.278886 (+ 197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.278889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.278890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.278903 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.280698 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.281217 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.281221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.281223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.281305 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.281600 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.281602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.281604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.281785 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.282552 (+ 767us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.282625 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.282732 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":229,"thread_start_us":113,"threads_started":1}
I20260926 07:29:35.282967 22904 log.cc:824] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:35.283818 22753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:35.282763 22727 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6944f46bd31e4564b63244b602da746e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56628
I20260926 07:29:35.284736 22753 log.cc:824] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:35.285452 22828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4: Bootstrap starting.
I20260926 07:29:35.286849 22753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e: No bootstrap required, opened a new log
I20260926 07:29:35.287801 22828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:35.288708 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305: No bootstrap required, opened a new log
I20260926 07:29:35.288681 22828 log.cc:824] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:35.289947 22753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } }
I20260926 07:29:35.290174 22753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6944f46bd31e4564b63244b602da746e, State: Initialized, Role: FOLLOWER
I20260926 07:29:35.290511 22753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } }
I20260926 07:29:35.290982 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } } }
I20260926 07:29:35.291224 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:29:35.291657 22828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4: No bootstrap required, opened a new log
I20260926 07:29:35.291870 22753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:35.291913 22727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:29:35.292012 22727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:35.292313 22904 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } }
I20260926 07:29:35.292508 22904 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa773fdbbce54de18cc405123c193305, State: Initialized, Role: FOLLOWER
I20260926 07:29:35.292831 22904 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } }
I20260926 07:29:35.293357 22727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } }
I20260926 07:29:35.293610 22920 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } } }
I20260926 07:29:35.293795 22920 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:29:35.294132 22904 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:35.294353 22727 leader_election.cc:288] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [CANDIDATE]: Term 1 election: Requested vote from peers d98e34c12c1d4ffd90e0439c60b31ad4 (127.0.98.61:41639), aa773fdbbce54de18cc405123c193305 (127.0.98.60:35117)
I20260926 07:29:35.295276 22878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6944f46bd31e4564b63244b602da746e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aa773fdbbce54de18cc405123c193305"
I20260926 07:29:35.295487 22878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:35.296840 22878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6944f46bd31e4564b63244b602da746e in term 1.
W20260926 07:29:35.297331 22706 tablet.cc:2369] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:29:35.297777 22702 leader_election.cc:302] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6944f46bd31e4564b63244b602da746e, aa773fdbbce54de18cc405123c193305; no voters:
I20260926 07:29:35.298200 22914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:29:35.298422 22914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [term 1 LEADER]: Becoming Leader. State: Replica: 6944f46bd31e4564b63244b602da746e, State: Running, Role: LEADER
I20260926 07:29:35.298717 22914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } }
I20260926 07:29:35.300328 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6944f46bd31e4564b63244b602da746e. Latest consensus state: current_term: 1 leader_uuid: "6944f46bd31e4564b63244b602da746e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } } }
I20260926 07:29:35.300438 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:35.297371 22685 master_runner.cc:186] Error getting master registration for 127.0.98.61:41639: OK, instance_id { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" instance_seqno: 1790407775017743 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:29:35.300580 22685 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 07:29:35.301275 22941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 07:29:35.301164 22828 raft_consensus.cc:348] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } }
W20260926 07:29:35.301353 22941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:35.301373 22828 raft_consensus.cc:729] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98e34c12c1d4ffd90e0439c60b31ad4, State: Initialized, Role: FOLLOWER
I20260926 07:29:35.301667 22828 consensus_queue.cc:260] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } }
I20260926 07:29:35.302208 22949 sys_catalog.cc:455] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } } }
I20260926 07:29:35.302436 22949 sys_catalog.cc:458] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 07:29:35.303661 22926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:29:35.303740 22926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:35.303815 22948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:35.304342 22828 sys_catalog.cc:564] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:35.304430 22802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6944f46bd31e4564b63244b602da746e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4"
I20260926 07:29:35.304669 22948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:35.304636 22802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:35.305887 22802 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6944f46bd31e4564b63244b602da746e in term 1.
I20260926 07:29:35.306387 22835 master_runner.cc:186] Error getting master registration for 127.0.98.61:41639: OK, instance_id { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" instance_seqno: 1790407775017743 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:29:35.310014 22835 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 07:29:35.313663 22961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:29:35.313748 22961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:35.317211 22802 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4 [term 1 FOLLOWER]: Refusing update from remote peer 6944f46bd31e4564b63244b602da746e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:35.317272 22878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305 [term 1 FOLLOWER]: Refusing update from remote peer 6944f46bd31e4564b63244b602da746e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:35.317941 22914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:35.321749 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.296714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.0.98.62:42121)
0926 07:29:35.296844 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.296851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.307843 (+ 10992us) server_negotiation.cc:407] Connection header received
0926 07:29:35.308305 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.308308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.308351 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.308401 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.310300 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.311045 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.312079 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.312409 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.312498 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.314259 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.314262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.314265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.314289 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.316135 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.316606 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.316610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.316611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.316703 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.317019 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.317022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.317024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.317220 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.317318 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.317795 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.317923 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:29:35.321897 22944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:35.322711 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.298361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42176 (local address 127.0.98.60:35117)
0926 07:29:35.300660 (+ 2299us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.300665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.308062 (+ 7397us) server_negotiation.cc:407] Connection header received
0926 07:29:35.308179 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.308182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.308226 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.308305 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.312301 (+ 3996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.313054 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.314240 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.314608 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.314717 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.318764 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.318768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.318771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.318800 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.320594 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.321076 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.321079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.321081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.321165 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.321473 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.321477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.321481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.321691 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.321828 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.322432 (+ 604us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.322556 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2226}
I20260926 07:29:35.322956 22949 sys_catalog.cc:455] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6944f46bd31e4564b63244b602da746e. Latest consensus state: current_term: 1 leader_uuid: "6944f46bd31e4564b63244b602da746e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } } }
I20260926 07:29:35.323099 22949 sys_catalog.cc:458] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:29:35.323802 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:35.300722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.0.98.61:41639)
0926 07:29:35.300825 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.300829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.303862 (+ 3033us) server_negotiation.cc:407] Connection header received
0926 07:29:35.304057 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.304060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.304105 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.304156 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.305435 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.306201 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.309363 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:35.309686 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:35.309762 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:35.313129 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:35.313133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:35.313136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:35.313163 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:35.315146 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.315626 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.315629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.315631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.315713 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:35.316002 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:35.316006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:35.316007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:35.316203 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:35.316318 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:35.323517 (+ 7199us) server_negotiation.cc:299] Negotiation successful
0926 07:29:35.323637 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:29:35.325728 22920 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6944f46bd31e4564b63244b602da746e. Latest consensus state: current_term: 1 leader_uuid: "6944f46bd31e4564b63244b602da746e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } } }
I20260926 07:29:35.325870 22920 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:29:35.335507 22944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6944f46bd31e4564b63244b602da746e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } } }
I20260926 07:29:35.335614 22920 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6944f46bd31e4564b63244b602da746e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } } }
I20260926 07:29:35.335677 22944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:35.335726 22920 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:29:35.335798 22944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6944f46bd31e4564b63244b602da746e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } } }
I20260926 07:29:35.335873 22967 mvcc.cc:205] Tried to move back new op lower bound from 7333510247693209600 to 7333510247626518528. Current Snapshot: MvccSnapshot[applied={T|T < 7333510247693209600}]
I20260926 07:29:35.335922 22944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:35.336165 22962 mvcc.cc:205] Tried to move back new op lower bound from 7333510247693209600 to 7333510247626518528. Current Snapshot: MvccSnapshot[applied={T|T < 7333510247693209600}]
I20260926 07:29:35.337282 22949 sys_catalog.cc:455] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6944f46bd31e4564b63244b602da746e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6944f46bd31e4564b63244b602da746e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42121 } } peers { permanent_uuid: "d98e34c12c1d4ffd90e0439c60b31ad4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41639 } } peers { permanent_uuid: "aa773fdbbce54de18cc405123c193305" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35117 } } }
I20260926 07:29:35.337399 22949 sys_catalog.cc:458] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:29:35.337651 22965 mvcc.cc:205] Tried to move back new op lower bound from 7333510247693209600 to 7333510247626518528. Current Snapshot: MvccSnapshot[applied={T|T < 7333510247693209600}]
I20260926 07:29:35.348624 22948 catalog_manager.cc:1347] Generated new cluster ID: 7ec507e3135047d59c99605b33deb81f
I20260926 07:29:35.348711 22948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:35.379833 22948 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:35.381208 22948 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:35.415405 22948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6944f46bd31e4564b63244b602da746e: Generated new TSK 0
I20260926 07:29:35.416339 22948 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:29:35.448171 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:35.445309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.0.98.62:42121)
0926 07:29:35.445435 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.445440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.447126 (+ 1686us) server_negotiation.cc:407] Connection header received
0926 07:29:35.447334 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.447338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.447389 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.447441 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.448038 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56658: BlockingRecv error: recv got EOF from 127.0.0.1:56658 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 07:29:35.449491 22910 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:35.445983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.0.98.60:35117)
0926 07:29:35.446092 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.446096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.448637 (+ 2541us) server_negotiation.cc:407] Connection header received
0926 07:29:35.448803 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.448806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.448854 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.448904 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.449377 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42184: BlockingRecv error: recv got EOF from 127.0.0.1:42184 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 07:29:35.450928 22834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:35.446546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.0.98.61:41639)
0926 07:29:35.446651 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.446656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.449852 (+ 3196us) server_negotiation.cc:407] Connection header received
0926 07:29:35.450235 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.450238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.450286 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.450337 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.450826 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33602: BlockingRecv error: recv got EOF from 127.0.0.1:33602 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 07:29:35.455250 22834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:35.453716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.0.98.61:41639)
0926 07:29:35.453826 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.453830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.454331 (+ 501us) server_negotiation.cc:407] Connection header received
0926 07:29:35.454517 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.454520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.454581 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.454628 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.455133 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33610: BlockingRecv error: recv got EOF from 127.0.0.1:33610 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 07:29:35.460054 22834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:35.458644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.0.98.61:41639)
0926 07:29:35.458740 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.458744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.459253 (+ 509us) server_negotiation.cc:407] Connection header received
0926 07:29:35.459422 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.459426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.459472 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.459518 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.459965 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33624: BlockingRecv error: recv got EOF from 127.0.0.1:33624 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 07:29:35.465744 22834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:35.464315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.0.98.61:41639)
0926 07:29:35.464404 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.464408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.464836 (+ 428us) server_negotiation.cc:407] Connection header received
0926 07:29:35.465009 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.465014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.465058 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.465107 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.465636 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33636: BlockingRecv error: recv got EOF from 127.0.0.1:33636 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 07:29:35.473488 22834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:35.471969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.0.98.61:41639)
0926 07:29:35.472097 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.472101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.472568 (+ 467us) server_negotiation.cc:407] Connection header received
0926 07:29:35.472761 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.472765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.472815 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.472867 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.473391 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33644: BlockingRecv error: recv got EOF from 127.0.0.1:33644 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 07:29:35.485493 22834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:35.483888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.0.98.61:41639)
0926 07:29:35.483999 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.484003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.484520 (+ 517us) server_negotiation.cc:407] Connection header received
0926 07:29:35.484730 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.484733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.484786 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.484841 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.485389 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33652: BlockingRecv error: recv got EOF from 127.0.0.1:33652 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 07:29:35.505833 22834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:35.504262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.0.98.61:41639)
0926 07:29:35.504388 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.504392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.504901 (+ 509us) server_negotiation.cc:407] Connection header received
0926 07:29:35.505085 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.505089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.505138 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.505189 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.505717 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33660: BlockingRecv error: recv got EOF from 127.0.0.1:33660 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 07:29:35.541694 22834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:35.540361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.98.61:41639)
0926 07:29:35.540529 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.540536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.540673 (+ 137us) server_negotiation.cc:407] Connection header received
0926 07:29:35.540814 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.540817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.540864 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.540940 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.541567 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33674: BlockingRecv error: recv got EOF from 127.0.0.1:33674 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 07:29:35.609102 22834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:35.607839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.0.98.61:41639)
0926 07:29:35.607971 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.607975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.608266 (+ 291us) server_negotiation.cc:407] Connection header received
0926 07:29:35.608374 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.608377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.608433 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.608506 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.608936 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33680: BlockingRecv error: recv got EOF from 127.0.0.1:33680 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 07:29:35.740531 22834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:35.739307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.0.98.61:41639)
0926 07:29:35.739474 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.739479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.739611 (+ 132us) server_negotiation.cc:407] Connection header received
0926 07:29:35.739785 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.739789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.739846 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.739940 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.740415 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33684: BlockingRecv error: recv got EOF from 127.0.0.1:33684 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 07:29:35.999907 22834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:35.998627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.0.98.61:41639)
0926 07:29:35.998777 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:35.998782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:35.998910 (+ 128us) server_negotiation.cc:407] Connection header received
0926 07:29:35.999127 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:35.999131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:35.999190 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:35.999298 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:35.999785 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33692: BlockingRecv error: recv got EOF from 127.0.0.1:33692 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:29:36.302593 22941 catalog_manager.cc:1259] Loaded cluster ID: 7ec507e3135047d59c99605b33deb81f
I20260926 07:29:36.302709 22941 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305: loading cluster ID for follower catalog manager: success
I20260926 07:29:36.304565 22941 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305: acquiring CA information for follower catalog manager: success
I20260926 07:29:36.305802 22941 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa773fdbbce54de18cc405123c193305: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:29:36.314836 22961 catalog_manager.cc:1259] Loaded cluster ID: 7ec507e3135047d59c99605b33deb81f
I20260926 07:29:36.314893 22961 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4: loading cluster ID for follower catalog manager: success
I20260926 07:29:36.316534 22961 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4: acquiring CA information for follower catalog manager: success
I20260926 07:29:36.317631 22961 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d98e34c12c1d4ffd90e0439c60b31ad4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:29:36.518375 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:36.513939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.0.98.61:41639)
0926 07:29:36.514214 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:36.514218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:36.514275 (+ 57us) server_negotiation.cc:407] Connection header received
0926 07:29:36.514428 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:36.514430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:36.514559 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:36.514659 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:36.514914 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:36.515586 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:36.516630 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:36.516905 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:36.517012 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:36.517365 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:36.517820 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:36.517929 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:36.518101 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 07:29:36.518152 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260926 07:29:36.525331 23043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:36.521526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.0.98.60:35117)
0926 07:29:36.521809 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:36.521813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:36.521826 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:29:36.521908 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:36.521912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:36.522028 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:36.522139 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:36.522386 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:36.522924 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:36.523906 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:36.524140 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:36.524227 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:36.524486 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:36.524909 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:36.525002 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:36.525077 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 07:29:36.525127 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
I20260926 07:29:36.532869 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:36.528454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.0.98.61:41639)
0926 07:29:36.528606 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:36.528610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:36.528912 (+ 302us) server_negotiation.cc:407] Connection header received
0926 07:29:36.529119 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:36.529123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:36.529181 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:36.529260 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:36.529598 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:36.530365 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:36.531658 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:36.531923 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:36.531980 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:36.532317 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:36.532402 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:36.532480 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:36.532642 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 07:29:36.532698 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:29:36.532960 23043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:36.528784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.0.98.60:35117)
0926 07:29:36.528901 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:36.528908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:36.528982 (+ 74us) server_negotiation.cc:407] Connection header received
0926 07:29:36.529144 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:36.529152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:36.529193 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:36.529272 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:36.529491 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:36.530300 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:36.531440 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:36.531679 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:36.531801 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:36.532499 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:36.532571 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:36.532637 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:36.532777 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 07:29:36.532827 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:29:36.534219 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:36.528390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.0.98.62:42121)
0926 07:29:36.529154 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:36.529159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:36.529174 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:29:36.529272 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:36.529276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:36.529448 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:36.529599 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:36.529918 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:36.530608 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:36.531789 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:36.532063 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:36.532147 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:36.532517 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:36.532877 (+ 360us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:36.532961 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:36.534054 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0926 07:29:36.534111 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":78,"threads_started":1}
I20260926 07:29:36.537420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 22685
I20260926 07:29:36.543766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 22760
I20260926 07:29:36.549959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 22835
2026-09-26T07:29:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3080 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:29:36 dist-test-slave-g04g krb5kdc[23054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:29:36 dist-test-slave-g04g krb5kdc[23054](info): set up 2 sockets
Sep 26 07:29:36 dist-test-slave-g04g krb5kdc[23054](info): commencing operation
krb5kdc: starting...
W20260926 07:29:37.587535 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/test-user.keytab.
Sep 26 07:29:37 dist-test-slave-g04g krb5kdc[23054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:29:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:29:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 07:29:37.706509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39841
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39841,127.0.98.61:44877,127.0.98.60:33995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:29:37.815434 23067 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:29:37.815729 23067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:37.815797 23067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:37.815865 23067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:37.819766 23067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:37.819830 23067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:37.819862 23067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:37.819912 23067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:37.819948 23067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:37.825098 23067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39841,127.0.98.61:44877,127.0.98.60:33995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:37.826072 23067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:37.827136 23067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:37.835341 23072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:37.835378 23075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:37.835579 23067 server_base.cc:1034] running on GCE node
W20260926 07:29:37.835376 23073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:37.836190 23067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:37.837177 23067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:37.838332 23067 hybrid_clock.cc:648] HybridClock initialized: now 1790407777838301 us; error 45 us; skew 500 ppm
Sep 26 07:29:37 dist-test-slave-g04g krb5kdc[23054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:37.840955 23067 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 07:29:37.841820 23067 webserver.cc:460] Webserver started at http://127.0.98.62:35815/ using document root <none> and password file <none>
I20260926 07:29:37.842356 23067 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:37.842446 23067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:37.842669 23067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:37.844703 23067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/data/instance:
uuid: "2fc7620961904bdcac63dbe7309036aa"
format_stamp: "Formatted at 2026-09-26 07:29:37 on dist-test-slave-g04g"
server_key: "a9e3cae45b23e71d17887dcc4bfd1aa8"
server_key_iv: "ffae38425acb34500ecbb99205ca26bf"
server_key_version: "encryptionkey@0"
I20260926 07:29:37.845264 23067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fc7620961904bdcac63dbe7309036aa"
format_stamp: "Formatted at 2026-09-26 07:29:37 on dist-test-slave-g04g"
server_key: "a9e3cae45b23e71d17887dcc4bfd1aa8"
server_key_iv: "ffae38425acb34500ecbb99205ca26bf"
server_key_version: "encryptionkey@0"
I20260926 07:29:37.848814 23067 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:29:37.851224 23082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:37.852499 23067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:29:37.852659 23067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/wal
uuid: "2fc7620961904bdcac63dbe7309036aa"
format_stamp: "Formatted at 2026-09-26 07:29:37 on dist-test-slave-g04g"
server_key: "a9e3cae45b23e71d17887dcc4bfd1aa8"
server_key_iv: "ffae38425acb34500ecbb99205ca26bf"
server_key_version: "encryptionkey@0"
I20260926 07:29:37.852785 23067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:37.881991 23067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:37.885151 23067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:37.885383 23067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:37.893052 23067 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39841
I20260926 07:29:37.893321 23134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39841 every 8 connection(s)
I20260926 07:29:37.894156 23067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/data/info.pb
I20260926 07:29:37.897433 23135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:37.901190 23135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:29:37.902525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 23067
I20260926 07:29:37.902673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-0/wal/instance
I20260926 07:29:37.902980 392 external_mini_cluster.cc:1428] Setting key 83c9e0ce7109cd373da257e661d73082
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 07:29:37 dist-test-slave-g04g krb5kdc[23054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:37.920787 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:37.903573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.0.1:54468)
0926 07:29:37.904159 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:37.904209 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:37.905039 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:37.905944 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:37.905952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:37.906438 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:37.906815 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:37.906836 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:37.908151 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:37.908158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:37.909477 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:37.909488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:37.910164 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:37.910169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:37.910365 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:37.910995 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:37.911021 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:37.915534 (+ 4513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:37.917937 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:37.917945 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:37.917954 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:37.918173 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:37.918546 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:37.918548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:37.918549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:37.918649 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:37.919474 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:37.919479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:37.919601 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 07:29:37.919740 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 07:29:37.920005 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":30,"thread_start_us":118,"threads_started":1}
I20260926 07:29:37.920795 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:37.903573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.0.98.62:39841)
0926 07:29:37.904152 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:37.904171 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:37.904331 (+ 160us) server_negotiation.cc:407] Connection header received
0926 07:29:37.905355 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:37.905363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:37.905655 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:37.905867 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:37.906995 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:37.908015 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:37.909612 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:37.909992 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:37.910139 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:37.915716 (+ 5577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:37.915721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:37.915732 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:37.915753 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:37.917806 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:37.918281 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:37.918290 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:37.918295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:37.918448 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:37.918744 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:37.918750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:37.918752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:37.919343 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:37.919486 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:37.919859 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 07:29:37.920087 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":440,"spinlock_wait_cycles":42240,"thread_start_us":171,"threads_started":1}
I20260926 07:29:37.923177 23135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:29:37.923949 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:37.923679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44877 (local address 127.0.0.1:54514)
0926 07:29:37.923798 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:37.923871 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:29:37.924187 23084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44877: connect: Connection refused (error 111)
I20260926 07:29:37.925182 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:37.924994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44877 (local address 127.0.0.1:54530)
0926 07:29:37.925092 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:37.925134 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 07:29:37.925390 23135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44877: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44877: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 07:29:37.932122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44877
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39841,127.0.98.61:44877,127.0.98.60:33995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:29:37.974629 23135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } attempt: 1
I20260926 07:29:37.975569 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:37.975271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44877 (local address 127.0.0.1:54532)
0926 07:29:37.975429 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:37.975490 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 07:29:37.976416 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:37.976180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44877 (local address 127.0.0.1:54540)
0926 07:29:37.976304 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:37.976359 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 07:29:37.976553 23135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44877: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44877: connect: Connection refused (error 111)
W20260926 07:29:38.042100 23142 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:29:38.042438 23142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:38.042505 23142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:38.042606 23142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 07:29:38.044718 23135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } attempt: 2
I20260926 07:29:38.045493 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.045203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44877 (local address 127.0.0.1:54554)
0926 07:29:38.045350 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.045421 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 07:29:38.046231 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.046017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44877 (local address 127.0.0.1:54564)
0926 07:29:38.046130 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.046179 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:29:38.046357 23135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44877: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44877: connect: Connection refused (error 111)
W20260926 07:29:38.047008 23142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:38.047101 23142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:38.047155 23142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:38.047200 23142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:38.047240 23142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:38.052393 23142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39841,127.0.98.61:44877,127.0.98.60:33995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:38.053457 23142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:38.054464 23142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:29:38.063377 23148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:38.063331 23147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:38.063336 23150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:38.063534 23142 server_base.cc:1034] running on GCE node
I20260926 07:29:38.064182 23142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:38.065290 23142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:38.066480 23142 hybrid_clock.cc:648] HybridClock initialized: now 1790407778066460 us; error 30 us; skew 500 ppm
Sep 26 07:29:38 dist-test-slave-g04g krb5kdc[23054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:38.069413 23142 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 07:29:38.070297 23142 webserver.cc:460] Webserver started at http://127.0.98.61:38709/ using document root <none> and password file <none>
I20260926 07:29:38.070885 23142 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:38.070978 23142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:38.071239 23142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:38.073231 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/data/instance:
uuid: "9fd76b0606db432a9cc8948350683de6"
format_stamp: "Formatted at 2026-09-26 07:29:38 on dist-test-slave-g04g"
server_key: "ca62b171a8b03ab494b312a1fa7b4aee"
server_key_iv: "90648614b0885a657a2e12fce81500dd"
server_key_version: "encryptionkey@0"
I20260926 07:29:38.073832 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/wal/instance:
uuid: "9fd76b0606db432a9cc8948350683de6"
format_stamp: "Formatted at 2026-09-26 07:29:38 on dist-test-slave-g04g"
server_key: "ca62b171a8b03ab494b312a1fa7b4aee"
server_key_iv: "90648614b0885a657a2e12fce81500dd"
server_key_version: "encryptionkey@0"
I20260926 07:29:38.077638 23142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 07:29:38.080814 23157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:38.082288 23142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:29:38.082402 23142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/wal
uuid: "9fd76b0606db432a9cc8948350683de6"
format_stamp: "Formatted at 2026-09-26 07:29:38 on dist-test-slave-g04g"
server_key: "ca62b171a8b03ab494b312a1fa7b4aee"
server_key_iv: "90648614b0885a657a2e12fce81500dd"
server_key_version: "encryptionkey@0"
I20260926 07:29:38.082515 23142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:38.096617 23142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:38.099627 23142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:38.099815 23142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:38.106398 23142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44877
I20260926 07:29:38.106738 23209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44877 every 8 connection(s)
I20260926 07:29:38.107369 23142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/data/info.pb
I20260926 07:29:38.107728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 23142
I20260926 07:29:38.107924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-1/wal/instance
I20260926 07:29:38.108244 392 external_mini_cluster.cc:1428] Setting key e0489b5b829a109ebe99388bd05160c4
I20260926 07:29:38.110123 23210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:38.113884 23210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 07:29:38 dist-test-slave-g04g krb5kdc[23054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:38.127619 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.115723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.0.98.62:39841)
0926 07:29:38.115880 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.115885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.116637 (+ 752us) server_negotiation.cc:407] Connection header received
0926 07:29:38.117593 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.117596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.117648 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.117744 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.119025 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.119583 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.120747 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.120987 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.121096 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.123491 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.123494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.123496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.123519 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.124958 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.125441 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.125446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.125447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.125533 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.125860 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.125863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.125865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.126038 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.126143 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.127344 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.127478 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 07:29:38.128000 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.116059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.0.1:54470)
0926 07:29:38.116475 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.116525 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:38.117399 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:38.117882 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:38.117887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:38.118359 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:38.118879 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.118893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.119708 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.119713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.120613 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.120622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.121127 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.121131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.121298 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.121881 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:38.121896 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:38.123320 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:38.125072 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.125078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.125091 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.125336 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.125623 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.125627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.125629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.125762 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.126141 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:38.126148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:38.126402 (+ 254us) client_negotiation.cc:769] Sending connection context
0926 07:29:38.126580 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 07:29:38.126912 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":114,"threads_started":1}
I20260926 07:29:38.129359 23210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 07:29:38.133981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_EJaZL/build/debug/bin/kudu
/tmp/dist-test-task_EJaZL/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33995
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39841,127.0.98.61:44877,127.0.98.60:33995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:29:38 dist-test-slave-g04g krb5kdc[23054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 07:29:38.137523 23135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } attempt: 3
I20260926 07:29:38.139343 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.129925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54570 (local address 127.0.98.61:44877)
0926 07:29:38.130239 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.130248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.130266 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:29:38.130302 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.130305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.130438 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.130584 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.130877 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.131730 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.132532 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.132870 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.132980 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.134891 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.134903 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.134920 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.134960 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.137283 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.137705 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.137716 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.137717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.137829 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.138163 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.138167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.138168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.138451 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.138581 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.138924 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.139104 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":219,"thread_start_us":148,"threads_started":1}
I20260926 07:29:38.139359 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.129824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44877 (local address 127.0.0.1:54570)
0926 07:29:38.130004 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.130019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:38.130100 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:38.130577 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:38.130580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:38.130701 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:38.130784 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.130788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.131825 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.131828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.132392 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.132400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.133005 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.133009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.133124 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.133558 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:38.133569 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:38.134763 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:38.137408 (+ 2645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.137411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.137412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.137586 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.137961 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.137967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.137979 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.138068 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.138555 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:38.138562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:38.138689 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 07:29:38.138847 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 07:29:38.139021 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 07:29:38.141800 23210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:29:38.142676 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.142295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36218)
0926 07:29:38.142495 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.142588 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
Sep 26 07:29:38 dist-test-slave-g04g krb5kdc[23054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260926 07:29:38.143486 23161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
I20260926 07:29:38.144565 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.144181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36224)
0926 07:29:38.144418 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.144484 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260926 07:29:38.144773 23210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33995: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
I20260926 07:29:38.146665 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.138019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44877 (local address 127.0.0.1:54584)
0926 07:29:38.138238 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.138255 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:38.138358 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:38.138701 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:38.138704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:38.138901 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:38.139097 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.139107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.140094 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.140097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.140797 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.140806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.141251 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.141255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.141349 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.141859 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:38.141873 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:38.143161 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:38.145252 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.145256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.145258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.145446 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.145731 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.145734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.145736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.145818 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.146168 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:38.146171 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:38.146271 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 07:29:38.146374 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 07:29:38.146512 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 07:29:38.146679 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.138115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.0.98.61:44877)
0926 07:29:38.138372 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.138379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.138392 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:29:38.138498 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.138503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.138549 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.138664 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.139225 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.139946 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.140915 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.141138 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.141239 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.143306 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.143309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.143311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.143330 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.145147 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.145550 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.145554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.145556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.145632 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.145905 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.145910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.145912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.146074 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.146177 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.146433 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.146531 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
I20260926 07:29:38.147433 23135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:29:38.148049 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.147830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36238)
0926 07:29:38.147938 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.147986 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 07:29:38.148720 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.148512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36248)
0926 07:29:38.148640 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.148679 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 07:29:38.148903 23135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33995: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
I20260926 07:29:38.180060 23135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } attempt: 1
I20260926 07:29:38.180846 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.180570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36258)
0926 07:29:38.180704 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.180770 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 07:29:38.181566 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.181353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36266)
0926 07:29:38.181457 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.181519 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 07:29:38.181715 23135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33995: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
I20260926 07:29:38.194042 23210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } attempt: 1
I20260926 07:29:38.194803 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.194546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36278)
0926 07:29:38.194682 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.194735 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 07:29:38.195549 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.195333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36288)
0926 07:29:38.195438 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.195496 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 07:29:38.195675 23210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33995: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
W20260926 07:29:38.250155 23216 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:29:38.250492 23216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:29:38.250571 23216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:29:38.250659 23216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:29:38.254613 23216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:29:38.254707 23216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:29:38.254750 23216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:29:38.254810 23216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:29:38.254850 23216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:29:38.256863 23135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } attempt: 2
I20260926 07:29:38.257690 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.257394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36304)
0926 07:29:38.257548 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.257616 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 07:29:38.258402 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.258200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36314)
0926 07:29:38.258305 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.258348 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 07:29:38.258522 23135 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33995: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
I20260926 07:29:38.260468 23216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39841,127.0.98.61:44877,127.0.98.60:33995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 06:33:24 UTC on ccycloud.quasar-gckabf.root.comops.site
build id 4673738
I20260926 07:29:38.261502 23216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:29:38.262521 23216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:29:38.263847 23210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } attempt: 2
I20260926 07:29:38.264623 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.264320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36316)
0926 07:29:38.264483 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.264544 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 07:29:38.265332 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.265118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36324)
0926 07:29:38.265229 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.265281 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 07:29:38.265460 23210 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33995: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33995: connect: Connection refused (error 111)
W20260926 07:29:38.271127 23223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:38.271234 23216 server_base.cc:1034] running on GCE node
W20260926 07:29:38.271134 23226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:29:38.271142 23224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:29:38.271838 23216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:29:38.272887 23216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:29:38.274057 23216 hybrid_clock.cc:648] HybridClock initialized: now 1790407778274042 us; error 28 us; skew 500 ppm
Sep 26 07:29:38 dist-test-slave-g04g krb5kdc[23054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:29:38.276772 23216 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 07:29:38.277596 23216 webserver.cc:460] Webserver started at http://127.0.98.60:40597/ using document root <none> and password file <none>
I20260926 07:29:38.278132 23216 fs_manager.cc:359] Metadata directory not provided
I20260926 07:29:38.278219 23216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:29:38.278440 23216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:29:38.280485 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/data/instance:
uuid: "412ebf5ea70847bab1acf34876120bf6"
format_stamp: "Formatted at 2026-09-26 07:29:38 on dist-test-slave-g04g"
server_key: "eb8d2804fb9d91707899569da03ce776"
server_key_iv: "52349dce8e65c1785a149ea95fda8044"
server_key_version: "encryptionkey@0"
I20260926 07:29:38.281038 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/wal/instance:
uuid: "412ebf5ea70847bab1acf34876120bf6"
format_stamp: "Formatted at 2026-09-26 07:29:38 on dist-test-slave-g04g"
server_key: "eb8d2804fb9d91707899569da03ce776"
server_key_iv: "52349dce8e65c1785a149ea95fda8044"
server_key_version: "encryptionkey@0"
I20260926 07:29:38.284581 23216 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 07:29:38.286918 23233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:29:38.288183 23216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 07:29:38.288321 23216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/wal
uuid: "412ebf5ea70847bab1acf34876120bf6"
format_stamp: "Formatted at 2026-09-26 07:29:38 on dist-test-slave-g04g"
server_key: "eb8d2804fb9d91707899569da03ce776"
server_key_iv: "52349dce8e65c1785a149ea95fda8044"
server_key_version: "encryptionkey@0"
I20260926 07:29:38.288425 23216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:29:38.310684 23216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:29:38.313876 23216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:29:38.314077 23216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:29:38.320719 23216 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33995
I20260926 07:29:38.321033 23285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33995 every 8 connection(s)
I20260926 07:29:38.321686 23216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/data/info.pb
I20260926 07:29:38.324345 23286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:29:38.327867 23286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:29:38.331225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_EJaZL/build/debug/bin/kudu as pid 23216
I20260926 07:29:38.331341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_EJaZL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790407489657243-392-0/minicluster-data/master-2/wal/instance
I20260926 07:29:38.331627 392 external_mini_cluster.cc:1428] Setting key c1a7022ed1b7bb5a52b37cb78a16cd5c
Sep 26 07:29:38 dist-test-slave-g04g krb5kdc[23054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 07:29:38 dist-test-slave-g04g krb5kdc[23054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 07:29:38.341744 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.329531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54484 (local address 127.0.98.62:39841)
0926 07:29:38.329669 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.329673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.330426 (+ 753us) server_negotiation.cc:407] Connection header received
0926 07:29:38.331464 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.331467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.331515 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.331560 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.332959 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.333512 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.334598 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.334803 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.334880 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.337784 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.337788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.337791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.337817 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.339579 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.340230 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.340232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.340233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.340310 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.340687 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.340689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.340690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.340866 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.340977 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.341457 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.341576 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:29:38.342638 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.332237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.0.98.60:33995)
0926 07:29:38.332592 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.332603 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.332629 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:29:38.332674 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.332679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.332829 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.332996 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.333683 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.334442 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.335509 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.335933 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.336014 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.338831 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.338836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.338845 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.338883 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.340696 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.341072 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.341078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.341079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.341161 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.341429 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.341433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.341435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.341647 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.341860 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.342058 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.342189 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":120,"threads_started":1}
I20260926 07:29:38.342645 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.329846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.0.1:54484)
0926 07:29:38.330283 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.330322 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:38.331188 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:38.331911 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:38.331917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:38.332314 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:38.332813 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.332825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.333646 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.333651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.334452 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.334461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.334943 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.334947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.335164 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.335763 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:38.335778 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:38.337637 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:38.339698 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.339706 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.339712 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.340117 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.340404 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.340410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.340412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.340590 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.340986 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:38.340995 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:38.341219 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 07:29:38.341402 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 07:29:38.341693 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":131,"threads_started":1}
I20260926 07:29:38.343933 23286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:29:38 dist-test-slave-g04g krb5kdc[23054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 07:29:38 dist-test-slave-g04g krb5kdc[23054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 07:29:38.352672 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.344390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44877 (local address 127.0.0.1:54596)
0926 07:29:38.344475 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.344488 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:38.344558 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:38.345093 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:38.345095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:38.345275 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:38.345399 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.345405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.346390 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.346393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.347199 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.347209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.347725 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.347727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.347798 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.348187 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:38.348202 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:38.349464 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:38.351272 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.351278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.351280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.351461 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.351741 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.351745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.351747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.351834 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.352237 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:38.352256 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:38.352354 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 07:29:38.352455 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 07:29:38.352566 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 07:29:38.352810 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.344650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54596 (local address 127.0.98.61:44877)
0926 07:29:38.344777 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.344782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.344795 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:29:38.344837 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.344840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.344892 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.344966 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.345530 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.346284 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.347304 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.347620 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.347719 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.349600 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.349603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.349606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.349645 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.351157 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.351560 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.351564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.351566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.351650 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.351922 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.351925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.351926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.352138 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.352244 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.352515 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.352641 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:29:38.353516 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.344886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.0.98.62:39841)
0926 07:29:38.344998 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.345001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.345012 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:29:38.345038 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.345040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.345069 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.345132 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.345777 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.346451 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.347463 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.347789 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.347875 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.350228 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.350230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.350232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.350251 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.351715 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.352111 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.352115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.352116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.352199 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.352470 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.352474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.352476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.352667 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.352865 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.353232 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.353343 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:29:38.353577 23286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:29:38.356612 23210 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } attempt: 3
I20260926 07:29:38.357663 23135 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } attempt: 3
Sep 26 07:29:38 dist-test-slave-g04g krb5kdc[23054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 07:29:38 dist-test-slave-g04g krb5kdc[23054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 07:29:38 dist-test-slave-g04g krb5kdc[23054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 07:29:38.363138 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.353987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36348)
0926 07:29:38.354118 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.354149 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:38.354223 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:38.354456 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:38.354462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:38.354632 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:38.354720 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.354726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.355593 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.355596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.356186 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.356196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.356741 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.356745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.356841 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.357301 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:38.357317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:38.358737 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:38.361238 (+ 2501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.361244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.361248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.361504 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.361802 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.361805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.361807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.361884 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.362175 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:38.362178 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:38.362259 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 07:29:38.362365 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 07:29:38.362937 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 07:29:38.363611 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.354125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.0.98.60:33995)
0926 07:29:38.354233 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.354237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.354250 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:29:38.354305 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.354307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.354345 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.354423 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.354832 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.355492 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.356302 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.356624 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.356728 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.358875 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.358882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.358885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.358915 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.361128 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.361615 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.361617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.361619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.361701 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.361956 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.361958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.361959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.362083 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.362183 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.363368 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.363478 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 07:29:38 dist-test-slave-g04g krb5kdc[23054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790407777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 07:29:38.366391 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.354662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.0.98.61:44877)
0926 07:29:38.354758 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.354761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.354808 (+ 47us) server_negotiation.cc:407] Connection header received
0926 07:29:38.354923 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.354925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.354963 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.355083 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.355719 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.356335 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.357333 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.357545 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.357655 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.360910 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.360914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.360916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.360943 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.363600 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.364838 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.364841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.364843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.364947 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.365293 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.365300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.365304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.365509 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.365696 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.366112 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.366230 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 07:29:38.367292 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6: Bootstrap starting.
I20260926 07:29:38.367960 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.356949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36350)
0926 07:29:38.357368 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.357384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:38.357457 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:38.358144 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:38.358147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:38.358324 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:38.358425 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.358433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.359816 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.359819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.360629 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.360637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.361212 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.361215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.361292 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.361681 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:38.361695 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:38.363227 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:38.366069 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.366072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.366074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.366666 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.367018 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.367021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.367023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.367140 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.367594 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:38.367598 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:38.367689 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 07:29:38.367769 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:29:38.367865 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325}
I20260926 07:29:38.368021 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.357990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33995 (local address 127.0.0.1:36356)
0926 07:29:38.358154 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:29:38.358167 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:29:38.358230 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:29:38.359430 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:29:38.359436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:29:38.359619 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:29:38.359723 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.359731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.360765 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.360768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.361536 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:29:38.361546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.362140 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.362147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:29:38.362262 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.362782 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:29:38.362793 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:29:38.364231 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:29:38.366398 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.366404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.366408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.366666 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.366984 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:29:38.366987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:29:38.366990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:29:38.367105 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:29:38.367594 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:29:38.367597 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:29:38.367687 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 07:29:38.367766 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:29:38.367885 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 07:29:38.369014 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.357008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36350 (local address 127.0.98.60:33995)
0926 07:29:38.357912 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.357925 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.357939 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:29:38.357970 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.357974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.358013 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.358829 (+ 816us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.358889 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.359680 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.360764 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.361088 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.361475 (+ 387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.363563 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.363566 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.363568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.363588 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.365960 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.366784 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.366787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.366789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.366902 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.367262 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.367265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.367267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.367462 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.368343 (+ 881us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.368421 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.368870 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":103,"threads_started":1}
I20260926 07:29:38.369326 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.358396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.0.98.60:33995)
0926 07:29:38.359232 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.359239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.359251 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:29:38.359280 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.359283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.359317 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.359397 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.359875 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.360635 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.361671 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.361996 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.362446 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.364372 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.364375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.364377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.364402 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.366289 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.366765 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.366769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.366771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.366865 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.367231 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.367234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.367236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.367483 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.369104 (+ 1621us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.369159 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.369232 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":80,"threads_started":1}
I20260926 07:29:38.370168 23286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:38.371100 23286 log.cc:824] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:38.371345 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:29:38.371770 23210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6: Bootstrap starting.
I20260926 07:29:38.373541 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6: No bootstrap required, opened a new log
I20260926 07:29:38.372495 23109 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2fc7620961904bdcac63dbe7309036aa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54494
I20260926 07:29:38.374229 23210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:38.375000 23210 log.cc:824] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:38.377003 23210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6: No bootstrap required, opened a new log
I20260926 07:29:38.377665 23286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } }
I20260926 07:29:38.377900 23286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412ebf5ea70847bab1acf34876120bf6, State: Initialized, Role: FOLLOWER
I20260926 07:29:38.378214 23135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa: Bootstrap starting.
I20260926 07:29:38.378213 23286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } }
I20260926 07:29:38.378762 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } } }
I20260926 07:29:38.379088 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:29:38.379563 23286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:38.380663 23135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa: Neither blocks nor log segments found. Creating new log.
I20260926 07:29:38.381052 23210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } }
I20260926 07:29:38.381254 23210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd76b0606db432a9cc8948350683de6, State: Initialized, Role: FOLLOWER
I20260926 07:29:38.381515 23135 log.cc:824] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa: Log is configured to *not* fsync() on all Append() calls
I20260926 07:29:38.381588 23210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } }
I20260926 07:29:38.382879 23210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:38.383471 23135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa: No bootstrap required, opened a new log
I20260926 07:29:38.383242 23216 master_runner.cc:186] Error getting master registration for 127.0.98.62:39841: OK, instance_id { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" instance_seqno: 1790407777890921 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:29:38.386261 23135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } }
I20260926 07:29:38.386461 23216 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:29:38.386461 23135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc7620961904bdcac63dbe7309036aa, State: Initialized, Role: FOLLOWER
I20260926 07:29:38.386730 23135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } }
I20260926 07:29:38.387198 23321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } } }
I20260926 07:29:38.387368 23321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [sys.catalog]: This master's current role is: FOLLOWER
W20260926 07:29:38.387446 23306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:29:38.387532 23306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:38.387923 23135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:29:38.388010 23109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:29:38.388135 23109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:38.389328 23109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } }
I20260926 07:29:38.386034 23142 master_runner.cc:186] Error getting master registration for 127.0.98.62:39841: OK, instance_id { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" instance_seqno: 1790407777890921 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:29:38.389948 23142 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:29:38.390255 23109 leader_election.cc:288] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [CANDIDATE]: Term 1 election: Requested vote from peers 9fd76b0606db432a9cc8948350683de6 (127.0.98.61:44877), 412ebf5ea70847bab1acf34876120bf6 (127.0.98.60:33995)
I20260926 07:29:38.390807 23184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2fc7620961904bdcac63dbe7309036aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9fd76b0606db432a9cc8948350683de6"
I20260926 07:29:38.391016 23184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:38.390969 23260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2fc7620961904bdcac63dbe7309036aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "412ebf5ea70847bab1acf34876120bf6"
I20260926 07:29:38.391201 23260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:29:38.392367 23184 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc7620961904bdcac63dbe7309036aa in term 1.
I20260926 07:29:38.392462 23260 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fc7620961904bdcac63dbe7309036aa in term 1.
I20260926 07:29:38.392759 23308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } } }
I20260926 07:29:38.392933 23308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:29:38.393224 23084 leader_election.cc:302] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fc7620961904bdcac63dbe7309036aa, 9fd76b0606db432a9cc8948350683de6; no voters:
I20260926 07:29:38.393554 23321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [term 1 FOLLOWER]: Leader election won for term 1
W20260926 07:29:38.393564 23332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:29:38.393654 23332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:38.393759 23321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [term 1 LEADER]: Becoming Leader. State: Replica: 2fc7620961904bdcac63dbe7309036aa, State: Running, Role: LEADER
I20260926 07:29:38.394025 23321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } }
W20260926 07:29:38.396296 23320 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:29:38.396368 23320 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:29:38.398270 23338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fc7620961904bdcac63dbe7309036aa. Latest consensus state: current_term: 1 leader_uuid: "2fc7620961904bdcac63dbe7309036aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } } }
I20260926 07:29:38.398403 23338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:38.399623 23067 master_runner.cc:263] No leader master found from master 2fc7620961904bdcac63dbe7309036aa
I20260926 07:29:38.399684 23067 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:29:38.407282 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.394717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.0.98.60:33995)
0926 07:29:38.394822 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.394827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.396188 (+ 1361us) server_negotiation.cc:407] Connection header received
0926 07:29:38.396378 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.396382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.396437 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.396497 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.399093 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.399920 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.401093 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.401468 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.401574 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.403566 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.403570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.403573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.403598 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.405391 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.405870 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.405873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.405875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.405960 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.406285 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.406289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.406291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.406498 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.406626 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.406970 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.407112 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:29:38.411181 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.395858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.0.98.62:39841)
0926 07:29:38.395977 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.395981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.397690 (+ 1709us) server_negotiation.cc:407] Connection header received
0926 07:29:38.397954 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.397957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.398035 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.398103 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.401114 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.401840 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.402936 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.403300 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.403377 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.407516 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.407520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.407523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.407547 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.409323 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.409834 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.409838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.409839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.409924 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.410242 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.410245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.410247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.410453 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.410547 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.410884 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.410995 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:29:38.413529 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:38.395736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.0.98.61:44877)
0926 07:29:38.395871 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.395878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.398079 (+ 2201us) server_negotiation.cc:407] Connection header received
0926 07:29:38.398239 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.398242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.398289 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.398344 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.399307 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.400009 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.407974 (+ 7965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:38.408282 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:38.408359 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:38.409985 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:29:38.409991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:29:38.409996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:29:38.410022 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:29:38.411722 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.412205 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.412210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.412213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.412294 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:29:38.412610 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:29:38.412613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:29:38.412615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:29:38.412816 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:29:38.412913 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:38.413264 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 07:29:38.413379 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:29:38.414202 23344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:29:38.415704 23344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:29:38.421494 23260 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6 [term 1 FOLLOWER]: Refusing update from remote peer 2fc7620961904bdcac63dbe7309036aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:38.422413 23338 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:38.426544 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fc7620961904bdcac63dbe7309036aa. Latest consensus state: current_term: 1 leader_uuid: "2fc7620961904bdcac63dbe7309036aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } } }
I20260926 07:29:38.426678 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:29:38.426985 23184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6 [term 1 FOLLOWER]: Refusing update from remote peer 2fc7620961904bdcac63dbe7309036aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:29:38.428905 23338 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:29:38.432989 23308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fc7620961904bdcac63dbe7309036aa. Latest consensus state: current_term: 1 leader_uuid: "2fc7620961904bdcac63dbe7309036aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } } }
I20260926 07:29:38.433100 23308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:29:38.436210 23338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fc7620961904bdcac63dbe7309036aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } } }
I20260926 07:29:38.436336 23338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:38.438570 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fc7620961904bdcac63dbe7309036aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } } }
I20260926 07:29:38.438699 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:29:38.438905 23345 mvcc.cc:205] Tried to move back new op lower bound from 7333510260405477376 to 7333510260304957440. Current Snapshot: MvccSnapshot[applied={T|T < 7333510260405477376}]
I20260926 07:29:38.440860 23346 mvcc.cc:205] Tried to move back new op lower bound from 7333510260405477376 to 7333510260304957440. Current Snapshot: MvccSnapshot[applied={T|T < 7333510260405477376}]
I20260926 07:29:38.448015 23321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fc7620961904bdcac63dbe7309036aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } } }
I20260926 07:29:38.448137 23321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa [sys.catalog]: This master's current role is: LEADER
I20260926 07:29:38.449144 23308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fc7620961904bdcac63dbe7309036aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fc7620961904bdcac63dbe7309036aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } peers { permanent_uuid: "9fd76b0606db432a9cc8948350683de6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44877 } } peers { permanent_uuid: "412ebf5ea70847bab1acf34876120bf6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33995 } } }
I20260926 07:29:38.449254 23308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:29:38.449963 23344 catalog_manager.cc:1347] Generated new cluster ID: 49b0cbc58d5d4b09a5483ff68e50adac
I20260926 07:29:38.450026 23344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:29:38.472687 23344 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:29:38.473949 23344 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:29:38.493498 23344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fc7620961904bdcac63dbe7309036aa: Generated new TSK 0
I20260926 07:29:38.494426 23344 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:29:38.525914 23292 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:38.523056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36378 (local address 127.0.98.60:33995)
0926 07:29:38.523201 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.523206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.524874 (+ 1668us) server_negotiation.cc:407] Connection header received
0926 07:29:38.525064 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.525067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.525117 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.525169 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.525781 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36378: BlockingRecv error: recv got EOF from 127.0.0.1:36378 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 07:29:38.527486 23138 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:38.523699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.0.98.62:39841)
0926 07:29:38.523807 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.523811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.526347 (+ 2536us) server_negotiation.cc:407] Connection header received
0926 07:29:38.526792 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.526795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.526841 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.526889 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.527370 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54508: BlockingRecv error: recv got EOF from 127.0.0.1:54508 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 07:29:38.528417 23217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:38.524308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54632 (local address 127.0.98.61:44877)
0926 07:29:38.524414 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.524418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.526533 (+ 2115us) server_negotiation.cc:407] Connection header received
0926 07:29:38.527720 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.527724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.527774 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.527825 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.528311 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54632: BlockingRecv error: recv got EOF from 127.0.0.1:54632 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 07:29:38.532694 23217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:38.531297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.0.98.61:44877)
0926 07:29:38.531391 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.531394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.531829 (+ 435us) server_negotiation.cc:407] Connection header received
0926 07:29:38.532025 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.532029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.532100 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.532150 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.532606 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54640: BlockingRecv error: recv got EOF from 127.0.0.1:54640 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 07:29:38.537426 23217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:38.536112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.0.98.61:44877)
0926 07:29:38.536205 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.536209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.536655 (+ 446us) server_negotiation.cc:407] Connection header received
0926 07:29:38.536819 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.536822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.536866 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.536913 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.537343 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54652: BlockingRecv error: recv got EOF from 127.0.0.1:54652 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 07:29:38.542696 23217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:38.541337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.0.98.61:44877)
0926 07:29:38.541425 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.541428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.541877 (+ 449us) server_negotiation.cc:407] Connection header received
0926 07:29:38.542062 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.542065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.542110 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.542156 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.542613 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54654: BlockingRecv error: recv got EOF from 127.0.0.1:54654 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 07:29:38.550137 23217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:38.548707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.0.98.61:44877)
0926 07:29:38.548830 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.548835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.549299 (+ 464us) server_negotiation.cc:407] Connection header received
0926 07:29:38.549474 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.549477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.549523 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.549572 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.550051 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54666: BlockingRecv error: recv got EOF from 127.0.0.1:54666 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 07:29:38.562047 23217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:38.560436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54670 (local address 127.0.98.61:44877)
0926 07:29:38.560544 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.560547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.561089 (+ 542us) server_negotiation.cc:407] Connection header received
0926 07:29:38.561281 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.561284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.561339 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.561388 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.561938 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54670: BlockingRecv error: recv got EOF from 127.0.0.1:54670 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 07:29:38.582386 23217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:38.580892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.0.98.61:44877)
0926 07:29:38.581033 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.581040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.581397 (+ 357us) server_negotiation.cc:407] Connection header received
0926 07:29:38.581624 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.581628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.581681 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.581736 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.582272 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54672: BlockingRecv error: recv got EOF from 127.0.0.1:54672 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 07:29:38.618144 23217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:38.616865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.0.98.61:44877)
0926 07:29:38.616995 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.617000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.617248 (+ 248us) server_negotiation.cc:407] Connection header received
0926 07:29:38.617396 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.617398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.617447 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.617555 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.618010 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54682: BlockingRecv error: recv got EOF from 127.0.0.1:54682 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 07:29:38.685660 23217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:38.684455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.0.98.61:44877)
0926 07:29:38.684591 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.684595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.684819 (+ 224us) server_negotiation.cc:407] Connection header received
0926 07:29:38.684961 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.684964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.685011 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.685095 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.685561 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54688: BlockingRecv error: recv got EOF from 127.0.0.1:54688 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 07:29:38.817207 23217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:38.815994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.0.98.61:44877)
0926 07:29:38.816160 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:38.816165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:38.816307 (+ 142us) server_negotiation.cc:407] Connection header received
0926 07:29:38.816483 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:38.816487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:38.816542 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:38.816639 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:38.817095 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54690: BlockingRecv error: recv got EOF from 127.0.0.1:54690 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 07:29:39.076294 23217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:29:39.075392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.0.98.61:44877)
0926 07:29:39.075522 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:39.075526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:39.075544 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:29:39.075684 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:39.075687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:39.075735 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:39.075830 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:29:39.076196 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54706: BlockingRecv error: recv got EOF from 127.0.0.1:54706 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:29:39.389024 23306 catalog_manager.cc:1259] Loaded cluster ID: 49b0cbc58d5d4b09a5483ff68e50adac
I20260926 07:29:39.389107 23306 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6: loading cluster ID for follower catalog manager: success
I20260926 07:29:39.390977 23306 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6: acquiring CA information for follower catalog manager: success
I20260926 07:29:39.392345 23306 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 412ebf5ea70847bab1acf34876120bf6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:29:39.397433 23320 catalog_manager.cc:1259] Loaded cluster ID: 49b0cbc58d5d4b09a5483ff68e50adac
I20260926 07:29:39.397490 23320 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6: loading cluster ID for follower catalog manager: success
I20260926 07:29:39.399185 23320 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6: acquiring CA information for follower catalog manager: success
I20260926 07:29:39.400386 23320 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9fd76b0606db432a9cc8948350683de6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:29:39.594170 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:39.590313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54718 (local address 127.0.98.61:44877)
0926 07:29:39.590556 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:39.590560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:39.590576 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:29:39.590697 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:39.590700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:39.590826 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:39.590937 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:39.591240 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:39.591812 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:39.592793 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:39.593025 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:39.593109 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:39.593365 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:39.593790 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:39.593896 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:39.593925 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 07:29:39.593972 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260926 07:29:39.600641 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:39.596956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.0.98.60:33995)
0926 07:29:39.597196 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:39.597200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:39.597214 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:29:39.597348 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:39.597351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:39.597477 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:39.597600 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:39.597808 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:39.598321 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:39.599168 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:39.599408 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:39.599512 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:39.599800 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:39.600220 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:39.600352 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:39.600439 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 07:29:39.600492 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":63,"threads_started":1}
I20260926 07:29:39.607896 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:39.603640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.0.98.60:33995)
0926 07:29:39.603904 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:39.603909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:39.604086 (+ 177us) server_negotiation.cc:407] Connection header received
0926 07:29:39.604344 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:39.604349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:39.604409 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:39.604516 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:39.604908 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:39.605693 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:39.606757 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:39.606997 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:39.607177 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:39.607369 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:39.607454 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:39.607548 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:39.607677 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 07:29:39.607731 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260926 07:29:39.608450 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:39.603629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.0.98.61:44877)
0926 07:29:39.603802 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:39.603807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:39.604085 (+ 278us) server_negotiation.cc:407] Connection header received
0926 07:29:39.604318 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:39.604324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:39.604384 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:39.604430 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:39.604876 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:39.605693 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:39.607099 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:39.607487 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:39.607601 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:39.607949 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:39.608060 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:39.608127 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:39.608248 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 07:29:39.608297 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:29:39.609513 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:29:39.603591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.0.98.62:39841)
0926 07:29:39.604221 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0926 07:29:39.604227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:29:39.604257 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:29:39.604434 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:29:39.604437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:29:39.604600 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:29:39.604720 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:29:39.604981 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:39.605735 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:39.607684 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:29:39.608087 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:29:39.608211 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:29:39.608699 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:29:39.609001 (+ 302us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:29:39.609133 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 07:29:39.609253 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 07:29:39.609317 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":83,"threads_started":1}
I20260926 07:29:39.612607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 23067
I20260926 07:29:39.619141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 23142
I20260926 07:29:39.624832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_EJaZL/build/debug/bin/kudu with pid 23216
2026-09-26T07:29:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3075 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6156 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (289961 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST